It never bothered me. Both of my babies were really cute and it made me feel good that they could bring a smile to people's faces.
One thing about babies -- they don't judge people based on physical appearances, the kind of car you drive, or the type of job you have. So, it's nice getting a smile from a human being once in a while, even if it is an infant.