My girls were 6 and 8 years old on our first trip. The decision was easy on the surface because we couldn't have been able to afford to go before then even if we wanted.
That said, we wouldn't have wanted. I found the money to go after they had reached that age. They could handle the walking, we could talk easily with them and share memories, no diapers, no naps and no special diets. It was intentional.
I know that many, and they will show up here soon, and tell you what a great thrill it was to see the faces of their youngsters and I don't put them down for that, but I do feel it is grossly overrated.

Why? OK, I'll confess here...I am selfish. If I am going to pay a lot of money to experience something, I want to experience it first hand. Not watch it through the eyes of an infant. Heck, they're fascinated with a flashlight. I wanted it to be a family experience that we ALL could remember.
25 years later I took them and their husbands and my 3 (at the time) grandkids to WDW. The rule still applied, as soon as the youngest was 6 years old we would go. Sadly, the newest member of the grandkid list will not have the same fortune. It was actually a once in a lifetime thing that I wanted to do. They youngest now is 6 months old, the second youngest is 10 years old. I'm getting way to old for this. And way to broke!
I do hope to still be around to go with her when she hits 6 but she will have missed the whole family gathering.