I smoke and go out 2 or 3 times a day. I've told my employees on many occassions that when they want a break, they should get up and take one! Everybody needs time away from their desk every now and then.
What I've noticed - and yes, I keep track because I'm a smoker - is that non-smokers that work for me actually do take as much time. Whether it's getting breakfast after the workday has started, or surfing the net, or making personal phone calls, etc. And in my office, smokers usually will skip lunch and eat at their desk where the non-smokers tend to go out for an hour. I think it does even out.
As for the smell, well, I've never had anybody complain about that. But I have had people in another department complain to me about one of my employees wearing too much perfume. If somebody complained about the smell of smoke, I'd handle it the same way. Everybody does deserve to be treated equally.