Some 4 year olds will do fine on ToT, and some 40 year olds hate it! It depends so much on the person. If the 4 year old is tall enough, okay with darkness, spooky stuff, and thrill ride type experiences, they will be fine. If not, they won't. My DD is 7, and she won't ride it- just seeing the height of the ride, and the spooky facade (and seeing and hearing the screaming riders) makes her stay away.
My best advice is, if the child doesn't want to ride, don't force him/her. The rest of the day, if not the trip, will be ruined- it will be practically impossible to get that child back on any other rides if they are scared out of their minds on one!!