I have not been on ToT for 3 or 4 trips. This is the only ride that DH will not ride since he does not like the drops. Because he won't ride it, I psyche myself out and think that it is too intense as well.
You can totally do TOT!! I have a similar story to yours. Near Toronto, we have a "theme park" called Canada's Wonderland and it also has one of these extreme drop type of rides, which I will NOT go on. I hate the feeling of leaving my stomach up above me.
However, on TOT, the drop feels much smaller than what I anticipated. I think the thing with TOT is that the theming is so absolutely wonderful, that it feels so spooky so a huge drop is not needed to get the scare factor.