My oldest was 4yo, middle child was 6mo, and youngest was 2mo.
Babies are very easy at WDW. It's when they get minds of their own and the motor skills to follow through with their ideas that it gets difficult- until the time they become able to reason more. Of course, bringing the babies was more about taking the older kids and us to WDW, but my babies really did love it. When my middle child saw Pooh for the first time, she squealed and wiggled and reached- clearly saying "give me my big, furry, fun moving toy!" She was the same for all the costumed characters- didn't like face characters until about 2-3yo...for her they were just weird looking strangers and she didn't do strangers.
If you're taking a "once in a lifetime" trip, I'd probably want my kids to be between the ages of 5-10. Not that my family could do that, as we have a 9 year span (currently a 15yo, 10yo and 6yo), but if you're going to visit more than once during their childhood I think anytime is a good time. They experience WDW differently at every age, and mine have loved it at every age.