My DD is 20 months old and has been to WDW three times: 4 months, 14 months, and 18 months. We weren't big Disney people before she was born, but we seemed to have become "Disney people" since she came into the picture!
The reason we went when she was 4 months old is because I wanted to take a vacation before I went back to work. We went back and forth about places to go and decided on Orlando (we didn't stay on property the entire time) because it was a family friendly destination and I figured that other travelers wouldn't freak out as much as a cruise or a more adult oriented resort if DD started to fuss. We (the adults) had a great vacation and my DD didn't have a bad time either!
We weren't planning the trip when she was 14 months, but the
free dining plan promotion came up and we decided that it was too good of a deal to pass up. We had planned to go the following fall (2006), but I am so glad we went when we did. All three of us had a great time! It took DD a day or two to adjust, but she got into it so much more than I imagined! We had such a good time that we went back when she was 18 months old and had an even better time (she loved, loved, LOVED the characters this time).
I get so frustrated when people look at me like I'm crazy when I tell them we've been to WDW three times in the last two years

. I get the "she (DD) won't remember it" comment all the time. Well, the people saying this just don't get it as far as I'm concerned! Of course she won't remember ANY trips that we take with her at this age, but we are not going to sit back and not travel until she is 3 or 4. We traveled a lot before my DD was born, and as long as we can afford to, we will continue to travel a lot with DD. We have just found WDW to be a very accomodating vacation destination to families with small children and we all have a good time when we are there. Some families go regularly on cruises or to all-inclusive resorts where you just lay around on a beach...we go to WDW!
Anyhow, to answer your question, I obviously don't think there is anything wrong with taking a baby. We they are really little, it is a very accomodating vacation destination for the parents and when they get a little older (but are still babies/toddlers), I think that they (and, as a result, you) will have a much better time than you would imagine.
Wow...I really rambled on...sorry!!!
