One thing that can make a difference in how comfortable you'll feel is your height. I'm 5'4" and would really have to work at it to fall out of a balloon basket. My husband, on the other hand, is 6' tall and was a bit uncomfortable the first time he flew because of the height of the basket sides. He got over it quickly, but it's something to consider.
I don't climb ladders and am scared of elevators, but have never been scared in a balloon. They feel very solid and if it bothers you, just don't look down. Looking out at the horizon gives you wonderful views.
Sheila