At 6 my son barely wanted to go inside the castle. He was just too spooked.
By the time he was 7 he would go on the ride. The dementors don't bug him but the "acromantula" do, so we have worked out a system where he knows the cue for when the spiders are about to show up (right after the dragon breathes fire), and I squeeze his leg to let him know when to open his eyes again (when you hear Hermione you know it's totally OK).
Of course, until we figured out the *hermione's voice* thing, it meant that I had to keep my eyes open! And I'm FAR more phobic than he is, LOL.