I would definitely GO!
We had a pool party for my son's 10th birthday. We had 7 boys there. My DH, dad, and another parent stayed at the pool while I did the food, cake, and other birthday duties. One of the boys jumped in and for some reason lost his breath and panicked. He grabbed one of the other boys down with him. I heard screaming and saw my hubby and dad jump in the pool to rescue the boys. The boys were so scared, but luckily they were okay. It scares me to think would could have happened if the men at the pool would not have been paying attention. It looked like horse play, but it wasn't.
Things like this happen so fast. We will NEVER have a pool party again for my kids. This was two years ago and I still get sick thinking about it. If my kids are invited to a pool party, I am right there!!!