Well, I wouldnt say I regretted it, but we had a tough trip when DS was 3.5. He was going through a stage where he had a hard time adjusting to new things. We went on the land/sea package and of course, everything was new(he had been to WDW at 15 months, but didnt remember much). He didnt want to do anything and we heard a lot of complaining. And example is when we went on Buzz, he had a blast "shooting Pop", and then with no line we were going on again and he refused to go. The entire trip was filled with things like that. By the time we were on the ship, I vowed I would never take him anywhere again, LOL! That didnt last long as we booked another trip for 11 months later. The good news is on that trip, he had the time of his life and loved everything. It was such a treat for me!
On the other hand, we have also gone when DS#2 was about 5 months shy of 3, and he didnt have any problem like that...however Im a little nervous of our August trip when he will be about the same age as DS#1 was during that bad trip.
I should add, I dont think any age is really too young. Between my 2 kids we have made trips at ages 8 mos, 15 mos, 20 mos, 2.5, 3.5, 4.5, 5.5. They are now 3 & 6 and we have 2 trips booked before their next birthdays. I think each age offers a new experience, and the chance of having a bad trip like we experienced is rare. Like I said, DS just happened to be in a phase