The wearing of hats indoors only by ladies is not sexist - it's old fashioned. And it continues today, in my opinion, because women don't like to admit they have "hat hair". I am a female so no flames please. Traditionally, men's hair is shorter so when wearing a hat and removing it, no one can see any difference in their hair style. But put a hat on a woman's head, and well, everyone can see the indentation in the hair.
Personally, I don't mind a man wearing a ball cap in a restaurant. Can't really think of any other kind of hat that men would wear at Disney World that would be totally offensive to someone.