While I consider myself quite a "rule follower", my initial reaction is along the lines of, "Why worry? Enjoy time with your friends." But in this case I think the fact that your fabulous family is larger than the family actually staying at the hotel makes it a bit too much.
If it were me, I might call the hotel and say, "My family is considering staying at the DLH in _________. Would it be possible to have friends join us for a swim while we're there? Is there a guest pass or something we could get?
This allows you to get the answer as if you were the person with the reservation, but doesn't put a burden on your friend to make the call.
I suspect others are correct and they'll say "no", but there's nothing wrong with a respectful question...