Yes it is OK to tell your kids not to go down there BUT speaking from experience it does not work. I live it too. Then you get into what about holidays & family get togethers? Are you planning on cutting them out completely? If not then the kids are confused. They will say why can't they go over there _______, but I have to go there for this event??? Then it turns into a power struggle with your Dh & his family. UGH, it really stinks!
A real solution for you would be to move. Otherwise I would just deal with it. Try and explain as best as you can why grandma & grandpa behaves in a certain way toward him. Sometimes it is time for the truth.
Do you see your parents? That is my saving grace, my mom.