Ok, I have to think about this...
DD17 was 6 months old the first time she went to Disneyland, 3 the first time she went to Disney World.
DD12 was 5 (only just) the first time she went to Disneyland, 7 the first time she went to Disney World.
DD8 was not-quite-2 the first time she went to Disneyland, and 4 the first time she went to Disney World.
Honestly, having experienced it with every possible range of ages, unless your child has an extremely easygoing personality I don't recommend it to anyone until a child is about 6. And they absolutely MUST be potty-trained, IMHO. I think our last trip, in 2010, when my kids were 16, 10, and 7 was by far the most enjoyable.
If you don't mind someone ALWAYS having to stay outside of rides with the baby, infants are pretty easy. They sleep a lot, and their food is pretty easy to maintain. You have to have a reclining stroller, a diaper bag, etc though and I always hated lugging all that crap along.