I agree with others, eating out several times a day, theme park tickets and souvies are not even close to a "little" something just for me because I work hard and I deserve it. It's hard to believe that your family could know about your financial situation and still think you should accompany them. Why would they want to put you further into financial insecurity?
A similar situation...my brother was supposed to accompany us on our Nov trip to Disney and my kids were thrilled about that. About 6 weeks ago, he told me that he had to back out because he just didn't have the money (and he was staying in our DVC unit only needing money for food, souvies and tickets). He racked up some serious CC debt a few years ago and is almost debt free. He had some unexpected car repairs come up and he just doesn't have the extra money to go on vacation. Although we are disappointed that he won't be joining us, I am glad to know that he has learned a valuable lesson and is working to get himself on a better path. We certainly understand his situation.
I can't imagine that your family or friend will be angry, upset or otherwise if they truly knew your financial predicament, and you explained why you just couldn't afford to join them on these trips. Money for food, clothing, shelter (and God forbid) medicine seems much more necessary than going on a trip that you really can not afford.
Think about what you NEED not what you WANT!!!!! I believe you have a very strong sense of entitlement, because your DH is a college graduate and making a decent living...but there are limits to everything. You need to change your way of thinking! My DH has an MBA and I have a bachelors degree. At this time, I am a SAHM and do without a lot so that I can afford to stay home. My children's education and needs come first. I have wanted to jet off to Aruba and Europe and Vegas in the past year or so and I can easily put those trips on a CC and go. That would mean that I have to get a job or put my kids in daycare and that is not something that I want to do, so that I could pay for the extra expenses.
You have to decide what is most important and I don't think you have done that yet. You will never be in the position that you want to be in until you decide a) what that is, and b) what you have to sacrifice to do it!