I totally agree with Puffkin. Have had a similar experience with both our boys at
Disneyland. They are NOW 5.5 and almost 4. The older one spent the entire year from 2-3 years old REGULARLY riding Matterhorn. He loved it. Since he turned 3, though, he is scared of EVERYTHING that is "big". He will only ride "small rides" (i.e. slow moving, not visually intimidating rides).
My younger one is a daredevil in EVERY sense of the word, BUT when we are at Disneyland, he is suddenly a chicken. To his credit, he will try new rides (as opposed to older brother who will make up his mind and stick to it). He has ridden Star Tours and sat like a champ, but the SECOND it was over, he couldn't get out of there fast enough. He said it was "too scary". Also too scary: SOARIN!!! He got freaked out by the fireworks at the end...too loud.

And PoTC. He is afraid of the dark, so this ride is torture for him. I took him a couple weeks ago (because last time he rode it he was around 2 and LOVED it). He sat in my lap with his head buried in my shoulder the WHOLE RIDE. He was also suddenly terrified of Toy Story Midway Mania a couple weeks ago. Can't tell you why. He has ridden it about 20 times, and all of a sudden, it was "scary".

I think it's a phase, since lately at home he's wanted to fall asleep with his light on (and the closet door open...I think he fears monsters in there).
So, you really need to know your kids, and don't push it. Make sure they have a full understanding of what the ride entails, and work your way up (like, don't start with ToT). There will be YEARS ahead where all they will want to do is ride the thrill rides, and you will be begging for mercy.
We are going to WDW in March. I think we will try Test Track, since DS4 LOVES cars. But, it's very loud, and that freaks him out too, so we'll have to see when we get there what he wants to do.