I haven't read the whole thread so if there is any additional info, I may be missing it. But, graduations, like weddings, don't happen often--once in a lifetime events. They rank above birthday parties, especially of a friend, imho, that happen yearly. I'm sure people would understand--friends child's 4th birthday party or sister's law school graduation.
If you are that worried about bringing your child, then leave DH at home, DS can go to the birthday party and you still do what, I really think, is MUCH more important. Whether or not you are "close", I still think it is important to be there. Besides, that closeness factor could change down the road (but probably not if you don't go).