We have a really similar setup to the OP. My parents live just up the hill from us on a dead end street. We are on the edge of town so not too many houses around. We moved here when my oldest DS was 3. Since we had horses around and I wasn't too thrilled with DS wandering all over the country side, I made DS call his grandparents before he left the house. So not only did he learn their phone number he could make sure they were home and had time for him. Then when he would leave the house the grandparents would call and let me know he was on his way home so I could watch for him. I've also had to talk to both DS's at times about knocking on grandma's door and not just walking in...so we do a courtesy knock then walk in and they do the same at our house.
Good luck OP, I think you need to discuss with your DS some rules for the grandson coming over. I think it's worked so well with us because we had the groundrules early on.