I suggest you delve a little deeper into the situation. By the time I was 6, all but one of my grandparents died. I do not have pleasant memories of the grandma that lived until I was 19. I can recall when I was little and my grandfather was still alive, her making some mean comments about me right in front of me and my grandpa defending me. Grandma was old as long as I can remember. She was 92 when she passed and I was 19, so she always seemed old, but I think it was her senility issues that made her act as she did. Either way, it was pretty damaging.
I guess what I am saying is, if you let this fester with your son, you may think he blew it off, but maybe he didn't. And maybe he never will. Get it out in the open.