Oh I feel for you. I have never had a friend "invite" themselves along but I do have allot of friends, acquaintances, and even family members in front of whom I will never be comfortable mentioning a trip. It really makes it hard to interact with them when you cannot share your excitement! It seems in my case that they always have something to say about my trips to WDW, and it is never anything nice. It usually involves some form of guilt over the fact that I am going and their family cannot, or their children have never been. I get allot of "it must be nice", and "but you don't even have children" ect. These are by no means impoverished folk- in fact most are more well off than I will ever be. They just have different priorities than my husband and I right now. We invest in travel, and they invest in way nicer things than I would ever care to own- not that there is anything wrong with that. I just don't go out of my way to make them feel guilty about their brand new "whatever".
I am leaving for WDW in 2 weeks and outside of this board the only people who know are my husband and a few select friends (the kind that know how to keep their mouths shut lol). It is not fair that I have to keep Disney "my dirty little secret" but it just works better that way. Oh they will find out as soon as I update my FB status and post pics, but this keeps them from destroying my pre-trip high lol.
If these people are the kind of close friends who you speak with on a daily basis, I would agree that it would be best to follow Flamingglobessigmunds advice, If/when they mention it, tell them how important that family time is to you and that you would love to help them plan a family trip of their own. There is no need to share any of your other fears over traveling with them, you don't want to hurt your friends feelings-(even though they have not been as kind to you

I believe this is called the High Road

)
Good Luck

and be sure to have an AMAZING trip , no matter what!