As a person I rarely wear hats, and I honestly can't remember when I wore a (non-winter) hat to even be able to take it off indoors.
As a waiter, however, I do have a sort of two cents on the issue. I do not have any problem with people who wear hats at my tables, be it a ballcap, cowboy hat, a womens dress hat, etc. However, there is one thing that is incredibly important to point out when wearing a hat, BE AWARE OF IT. It is more difficult to communicate with someone when you are sitting down wearing a hat and they are standing up. Eye contact and facial gestures are much less noticeable, your face is blocked by the visor as is your view, so if you are not very polite/actively paying attention and willing to look at me when I speak it makes taking the order, clearing and refilling items, etc much more difficult. This goes along with a lot of other factors, but I have to say probably 4/5 people who wear ballcaps inside do none of these things, and become a massive hassle. They also rarely tip, but boy am I not getting into that conversation here.
The point is: wear a hat if you want so long as you are polite and spatially aware and those of us that may actually be affected by it (as opposed to other people eating around you who likely are not) will not blink or think twice.