I would have waited for the family to arrive and had to find other seats, and then moved. But that would be the evil in me coming out.
Couple of ideas to help alleviate the issue:
As for the loungers, when we went I would sit by the pool, read and watch my DD. I had a couple of kids come by to swim (about 10), I asked them if they were going to be using the lounger they put their stuff. They said no, so I told them to put their shoes under my chair and keep the other chair freed up for someone who would use it. They were fine with that, and the open lounger was taken within minutes.
THe other thing we will do is take turns on the lounger. My BIL's girlfriend and I will rotate, since we are both fair skin we need to get out of the sun some. We would also eat lunch at different times also, that way the chair was always being used.
Also when my DD would get out of the pool, she would sit at the foot of the lounger, instead of getting her own. Usually she would eat a snack and be back in the pool within 15 minutes.