I think it all depends on how frequently you go. If Disney is a once in a lifetime trip, it is probably best to go when the kids are a little older, 8 to 12, so that they remember it better.
But if you plan on going frequently, every year or so, every age is totally magical and worth the trip. Even with an infant.
Personally, I loved taking my 3 and 4 year olds the best.