I think a lot of people question taking a young child because their thinking is that Disney is a once-in-a-lifetime trip. Many of us here are really lucky; we have already taken multiple trips and plan to take many more. Personally, I think every age at Disney can be magical, and once that age passes, that special kind of magic for that age passes, too. I like to take my girls every year because I don't want to miss any of the magic.
There are a lot of people, however, who really will only take the once in a lifetime trip, and they try to determine the perfect age to take their child/grandchild. My coworker was just saying something similar. She has a 3 year old grandson and was telling me how much he loves Cars. I suggested she take him to stay at Art of Animation, Cars section. She said, "He's not going to remember it if we go now. And it is just so expensive. I don't think we can afford it." To me, that said that she would love to take him, but she is waiting for that perfect magical age when he will remember the trip. I don't think there is anything wrong with that. She is single and supporting herself on her teacher's salary and trying to plan for retirement. She is hesitant to take him at age 3, because if they can only afford to go once, she wants him to remember it.
I'm never bothered when people make comments about taking young children to Disney. They have their reasons. I just take it as an opportunity to expound on how fabulous Disney World is and to talk more about my favorite place.