I think you really have to do two things when deciding how young is too young to go on a cruise:
1) How does your child deal with new environments, and how does Mom and Dad deal with their child in a new environment?
2) How do you plan to spend your time? Is this a Family vacation where you want to spend the time with each other, or is this a Family vacation where you all just do what you want, including being on you own?
Our first cruise was a 3-day on the Wonder with our then 27 month old DD. She was not potty trained, and she was too young to be in the clubs. We all had a great time. We took Swim Diapers with so she could be in the pool. We played on the beach at CC. Fish nibbled her little toes. She napped when she got tired, but we found you can't force your child to conform to his/her normal schedule on Vacation. Kids want to see and do everything (just like Mom and Dad), and just like Mom and Dad, they sometimes over do it. However, we also learned that kids are highly adaptable (at least ours proved to be). She was never at Flounders; she was always with us where ever we went. However, we went on this cruise knowing this. And personally, that's how we want it.
Now, just because she was with us all the time, doesn't mean we didn't get to do any "adult" activities. I had a massage, my wife had a spa treatment, and one night we actually spent some time on deck after our little one fell asleep in the cabin. It wasn't long, but Mom and Dad did get some alone time.
Anyway, you have to know your kid and make the decision. But I think trying to make you kids conform to your normal schedule is only setting you and your child up for a massive amount of stress and a stessful vacation. Know ahead of time what to expect, and you
will have a great time, regardless of your kid's age.
Just my 2 cents.
Oh, and I think a toddler can remember experiences like this. I know that our daughter remembers things that happend on the cruise that we do not have pictures or video of. Go for it! Enjoy the moment! Vacation is Vaction!