Let me give you the short story of what happened the last time I had some unintended guests on one of my trips. First my boyfriend at the time invited them not me, and the weasel wasn't even going himself. My best friend, who was invited, got her brand new wheelchair run over by a mack truck, we had to wait for someone to bring another to us on the road. In the meanwhile the rental car's battery died because I left the electric cooler plugged in while I waited. Someone in our group wrote a bad check at a hotel we were staying at, threw most of our camping equipment out on us while I was calling his mother on the phone to get her to pick him up, tried to kidnap my son and steal someone's bike,

the police were called and that individual was arrested, and sent on a bus back to NH. Other's in our group complained how long it took for me to drive down there, get us to the parks and back to our campground, never paid me a cent for their share of the trip, the only thing they paid for was their tickets into the park. My bff feet got severely bitten by red ants and the last night we were there I spent in the ER with her the entire night, and had to drive back to NH with no sleep, I was the only driver and drove for 24 hrs straight through on the way back, as I had to rent a car to fit all my "friends" in and it was due back the next day.
Now I know this was extreme, and it was when I was much younger, Ive certainly learned my lessons, on who to invite and not to invite on trips. However, if you really don't want these people to go just say very nicely "Gee, I would love for you to go but you know how hectic regular life can be and I really feel that this is time for me to spend with my family alone. It will allow us a chance to catch up a bit as we will finally be able to slow down from real life. Maybe we can plan a trip together sometime in the near future."