We first took my DS when he was about 16 mos old and he cried when he got OFF. We had to ride it about 4 times that first trip. When we would go back (every 3 or 4 months, we live close by), the first thing he would want to do was see the pirates. One of his first words was Pirate (which he pronounce "piway" at the time).
THEN, when he was about 2 1/2, he suddenly developed a fear of the dark, and now he refuses to go on the ride (he just turned 4, and still loves pirates, but won't go near that ride, or the HM, or JII, or anything that blacks out even for a few seconds).
So I guess you have a better chance of getting through it with a toddler than a 2-4 year old.