My kids were 7 and 10 before I took them, and they loved it. But that was only the 2nd time I'd been, so we got off to a bit of a late start.
Now that I've been going to Disney more frequently over the past several years, and know a bit more about the whole Disney experience, I'd have to say, for me personally, I wouldn't take a child under the age of about 4 or so. I would want them to be complete potty-trained. I know most kids are probably trained by 2 or so, but still, even after that, mine still had accidents now and then.
For me personally (again) I would think that taking an infant would be more trouble than it's worth, with feedings, bottles, diapers, etc. But that's just me. I know there are many many people who do go with infants and toddlers, and have a blast.