My kids are not easily scared and rode both HM and POTC as toddlers and on to present age. Whenever I would notice any apprehension, I'd whisper in their ears, "Remember, it's just 'Mickey Magic' and Mickey won't do anything to hurt you. It's all pretend," and they were fine.
We went last year with my cousin and her son, age 4, and he was much more timid than my kids were at that age, but he still rode both Pirates and HM. Pirates scared him a little at first, but he was OK by the end, and I don't think HM bothered him at all.