I find the biggest thing that separates us from our child-full friends is that we talk about our pets, and they talk about their kids.
But, on a more serious note...
A lot of people I know/meet are cool with our attitude about it all, but I do get the occasional instant dislike from a mommy type.
The thing is... I know that it's not a popular opinion, but I really just don't like kids. I never have, and I likely never will. I love love animals, esp. baby ones, but I have little tolerance for (in no particular order) kids, too much noise, stupid people, and rude/inconsiderate people.
I can occasionally handle a child or two on their own, if they are intelligent and well-behaved, or can't talk yet and aren't crying, but I much prefer the company of adults.
I usually keep that opinion to myself and try to temper it around people who are parents (or stupid/rude people I have to deal with like coworkers or family), though.