That little group sounds charming.

It's okay for them to say whatever they want to you, but you have to watch what you say because they'll complain? That's nuts. I'd have a hard time keeping a low profile around people like that, as in one way.
You do have a good attitude about it. You can't change them. Just limit the information they get. When you hear them talking about you and/or your hours I'd be tempted to say, "That's between me and the boss. That's none of your business. But if you'd like to share your business with everyone feel free.". I'd be tempted, don't know that I'd do it.