OP, I wanted to start by telling you I think you have some amazing sons. So many children and especially teenagers these days exclude anyone who is different, the fact that your boys befriended this young man to the point that they invited him to graduation and to join the family for birthday dinner speaks volumes about their character, and the way you've raised them.
I have very mixed feelings about the current fad of "trusting your instincts" as a way to judege whether or not it's safe. I've never read the book that it seems to come from (Protecting the Gift?), but in my experience human beings (and other animals) have a very strong instinct to be distrustful of that which is unfamiliar or different. While I do think that sometimes that instinct can be protective (that is we feel fearful of someone without knowing why because we've detected very subtle "differences"), it also leads us to be subconsciously fearful of people who differ from us in terms of race, or culture, or sexual orientation, or disability. I once had a conversation with a parent who was telling me how their child was fearful of certain people they met at the park and they taught their child to respect that fear -- except the child was fearful, it turned out, of every black person they met, and the parent was excusing rudeness or exclusionary behavior towards them.
If I were in your situation I'd make more efforts to get to know this young man, invite him to dinner. Include him in events where your family participates as a whole (e.g. maybe you're going on a hike or a trip to a museum other outing somewhere, and you want to invite him along), so that you and your husband can watch him interract with your son.
In addition, I do think it's appropriate to set some limits for him. While I don't see any red flags that his behavior is unsafe for your sons, I do think that young men with disabilities are at a huge risk for people thinking that they might be a molester and the repurcussions could be huge, so I could see saying to him "Jim, I know you're a nice person, and I know I can trust you. However, you need to know that there are parents out there who are very protective of their children, and I'm worried that someone will get mad at you. Let me tell you some rules that I think would be good rules for you . . ." and talk about appropriate limits like not showing up before school, not staying after dark, making sure he's met the parents of any kid he hangs out with etc . . .
Finally, if you really want to help, maybe think about things he can do to connect him to more of the community so he's not always on the fringe. He likes to watch kids play sports -- there's another thread here about how people are desperate for people to staff the stands at Little League -- maybe he could volunteer for that (or charge people -- I'll do your shift for $10 an hour!). Maybe he could join a local bowling league, or volunteer with the young kids at special olympics, or spend time at a nursing home helping out.