We waited to go until our son was 2.5 years old. We enjoy WDW, but we love to do a lot of different types of vacations, so we wanted to wait until he was old enough to start to have a few memories of it. He loved it so much, and we loved seeing how much he loved it, and we started going back about every 9 months. (That's when we bought into DVC, since we realized we were going often.)
With our daughter, we already had a trip planned with friends when we adopted her, so we ended up taking her when she was about 5 months old. Honestly, if we hadn't already had a special trip planned with this family, whose daughter had never been, we'd have canceled and waited until DD was a little older. I worried about the vaccinations she hadn't had. (One of my nephews almost died of whooping cough as an infant, before he'd been able to be vaccinated, so I'm very aware of how deadly infectious diseases can become.) We've gotten very sick at WDW on previous trips, despite the handwashing precautions we take, so, yes, I admit I was a little worried. It turned out to be a great trip, although it was a lot more work with a baby.
I am a big believer in travel helping to shape your perspectives on life. In general, I tell people to go for it, take your kids on lots of vacations (including WDW, but to all over). Our just turned 4 year old year old daughter (she of the 5 month old trip to WDW) has been with us on 3 cruises (to the Caribbean, Bahamas, and Central America), 4 or 5 trips to WDW, a trip to Hawaii and California, and weekend trips around Texas. Our kids LOVE to travel, and we've worked to make them good little travelers and instill the same passion for it that we have. We started with our son on cruises instead of at WDW; we thought cruises with a baby were much easier than WDW.