I don't think three times a year is so much to ask. Why do you hate it so much? I wish I had a large family like that
Since you don't have a large family you probably have a wonderful, close image of a big family that gets along,

like the Waltons, or a Norman Rockwell picture, or a Hallmark card type of family.

(Not at all flaming you for that.)
Unfortunately, the OP sounds like her DH's family is anything but that. Even HE doesn't want to go. In fact he feels quite guilty and obligated, so that tells us right there that this is a well established emotional pattern that goes on in his family.
As dngnb8 said in his/her?(?) post, he comes from a big Italian-Catholic family. This sounds like a gross stereotype, until you've actually spent a holiday with Italian-Catholic family,

but some types of families can be heavy on the drama dramas
, the trauma dramas

, the long suffering victims and martyrs who you have to listen to all the stories about (which are never ending

) and hold their hands.
(Not limited to Italian-Catholic families.

)
While those situations may be bearable, the worst problems are with the nosy, dysfunctional families where, the fact that you are all related somehow gives them the
right to totally overstep emotional, mental and moral boundaries. It's at events like these where just showing up, means they get ammo to talk about YOU amongst themselves first,
then, like a tribal committee, to share their unsolicited suggestions, advice and criticisms with you.
The fact that you are there, where they can SEE you, arms them with stuff to talk about,

from your hair, your weight gain, your age, your job, your lack of a job, your money, your
Disney vacations, your children and how you are raising them.
You spend your time either standing up for yourself, which they collectively see as hostile and arrogant. Or that you are
overly-defensive, in which they take as being right about you, all along.
You can't win. And you really can't hide amongst the crowd, unscathed. Some relatives actually look forward to these events
to bash the other relatives.
They are like heat seeking missiles, going after their known targets.
For some people, holidays are synonymous with family ordeals
instead of celebrations.
Not sure what is going on at the OP's holiday events, but the fact that her DH doesn't want to go either, but feels he HAS to because the repercussions would be worse, say a lot.
