We've gone every year since my son was 6 months and there are challenges with every age. From ages 1-3 they don't want to be in the stroller so you're in danger of losing them in the crowds, they're afraid of a lot of attractions and they don't always want to nap with all the excitement around them so they get cranky by 4pm; at 4-6 they can go on most things, but maybe not all things, so there's disappointment as well as the over-tired melt-downs, and they are still afraid of a lot of things; at 7-12 they're a bit easier again, but still have their issues, and by the time they're teens they might be bored with the whole thing, don't want to hang out with Mom and Dad, and have some definite opinons about what they will and won't ride.
It's life. The good generally outweighs the bad with any age, so if parents are up for the challenge I say, go for it!