The youngest I've taken a child s 7 months. I think any age is fine to go to WDW, but you will have a different trip at each age--speaking from experience.
At 7 months, we did not have the baby in character lines, but when characters were out and about they would stop and pay a ton of attention to my child. The baby slept a lot and could have cared less what was going on around him, but was fascinated with all there was to see. I have some beautiful memories and photos. We took advantage of child swap to do many rides and took the little one on tamer rides. We really enjoyed the trip and found him easier to handle at WDW than at home--he was needy and difficult.
At 15 months my kiddos enjoyed all that went on around them. One loved the characters; another loved them from a distance. Each of them came home with a love for anything Disney, although they never understood where that came from. Again, no worse than a day at the mall or anywhere else.
At 2 years the kids loved what they saw and believed in it all. They did not comprehend or appreciate all that WDW had to offer, but they loved the trip and were able to experience most of it.
At 3, it is truly magical. It is all real and all appreciated. My kids at that age enjoyed that trip the most. Very little wasn't experienced and the characters were enjoyed an extra amount.
At 4, my kids knew that things were more an illusion than real, but get caught up in it. They love what is going on, but definitely have opinions on what they should do and when. They are also a little selfish and don't realize that it is about the entire family. Ditto for age 5 and 6.
At 7, they start to appreciate the subtleties of it all.
It also depends on what your tolerance is. If you are the type that will avoid the mall or a restaurant with a young child, you'll find WDW challenging. If strollers and diaper bags are part of your daily routine anyhow, you won't notice any extra work with a child that age as you would any day trip.
I've been just about every year since 1976. I don't remember most trips so waiting until they can remember it seems silly to me. I have walked away with impressions and experiences that carried into the rest of my life and that is what I hope for/want for my kids. As an adult, I remember. And I have pictures from all trips. I go for me and my kids are just lucky benefactors of that.