Wow, I'm really impressed by generally how respectful this conversation has been, even with disagreements.
OP--new relationships at that age are tricky to navigate. You know if there are deeper issues going on regarding the boyfriend/SO. We can speculate to what they are, but odds are that you know. Whether you're caught in the middle or tied up in it yourself, just be grateful that you still have a relationship with your son and recognize he's navigating a tricky time in his life. Don't expect him to put himself through hostility just to spend family time with you--whether that is coming from yourself, his dad, brother, an uncle, whatever. Hostility can be as personal as not being invited to a wedding and as general as listening to someone spout on about politics you disagree with. It may not be directed at you, but it's tiresome.
I'd also add that the limited time frame is very likely for his boyfriend/SO's sake. It's great that you find meeting people so easy, but it's very draining for others. What some family consider to be warm and welcoming, is really not so much to others. My husband's family is generally much colder and more stand-offish than my family, and I feel uncomfortable around them even though they are very kind to me and have given me very few reasons to feel that way. They think they are very warm--and in many ways they are--but their nonverbal cues and signals are cold and very distracting to me. I get exhausted pretty quickly around them because I never feel quite at ease. My husband feels very overwhelmed around my family in longer doses, too, especially when we're all together around the holidays. He's not used to the intensity of it all. I think planning to see family in 3-4 hour chunks is a great idea, especially around the holidays. If things are going great, you can stay longer. If not, expectations are already in place and you can leave easily when you are too tired. It's possible they spend more time with his family because they find it easier. Maybe your son is more adaptable than his boyfriend/SO. That's not something to be hurt over necessarily.
Keep trying to make plans with your son in small doses. As others said, whirlwind trips are not fun and relaxing for everyone. Try to make plans for brunch or lunch, keep it light and simple. You're lucky that you live close enough that this is possible. Nothing is more frustrating than having family that makes small insinuations that you're not doing enough for them, when the efforts that you are making are more difficult than they realize. Have patience and love.
(Oh--and the response re: college loans vs. credit card debt sounded a wee bit telling. Being judgy about your kids' finances makes them feel horrible, NOT inspired to do better. It doesn't matter what kind of debt it is, just let go of your feelings about it.)