I think it depends on whether you are concerned about the child
enjoying it or concerned about the child
remembering it. It also depends on whether or not you plan on taking your child only once.
If your main objective is for the child to enjoy it:
A child as young as 6 months old will probably enjoy it because it is so stimulating (the colors, the lights, the sounds). So, really anytime is going to be enjoyable to the child, IMO. But may be hard for the parents (stroller, diaper changes, nap times, etc). I, personally, wouldn't take a small infant, but only because it would be more work and less fun for me. But, many people have done so and reported having a great time. My opinion also has to do with the fact that we only have gone in the summer months, that heat and a baby...not my idea of fun.
If you are concerned about them 'remembering' the experience:
You may as well wait until they are 7-10 years old. Any younger and they will only remember bits and pieces. (even at 4 yo: 2 of our children experienced WDW at 4 and remember very little of those visits) But, really...who cares, it is still a great time!
I, personally, think it WDW is a destination that requires multiple trips to really enjoy the full benefits. For us, WDW is an ongoing family vacation tradition. Collectively, my children have been: (5 months gestational

) 2, 4, 6, 7, 8, 10, 11 and 14 on our trips so far. They all have gotten something different out of each trip. Heck...we have gotten something special out of seeing them enjoy it at different ages! None of the trips were wasted. Not by a long shot.
But I will say...(now that my children are 10,14 and 17)...it sure was CHEAPER when the were younger, LOL!!!!
Edited to add: I notice you plan on multiple trips...keep in mind that you may have another child by 2007. If so, you are back to the same dilemma. I say go when YOU feel it would be a fun family time...no need to wait for the children to reach an 'optimum age'. (btw, he is FREE until 3!!!) I don't think there really is a 'perfect' time. Every trip is a great time! Also remember...the time FLIES by, it was only yesterday that my eldest was in a car seat...she will be a senior in HS next year! So, forget the 'rules' and have fun! JMHO.