I don't think there is any problem with it, especially during daylight hours. I think the 8pm cutoff at trader sam's is probably a good cutoff in general at disney for that sort of thing TBH. My son loves doing "grownup" things too and would love to sit and have a drink with mom (non alcoholic for him of course, lol). Sit, have a beverage, enjoy a little magic and then have an ice cream sounds good to me.
No one should be doing anything inappropriate for a kid to be around at 4pm in a disney bar