Well....first off.....if half of them don't swim, why even bother w/a pool party???? This is a dangerous proposition if the kids are not confident swimmers.
That said, I would suggest something ELSE to do than a pool party...ie, how about a sleepover where they could all do thier hair and watch a movie? or how about some happy meals and playtime? or maybe going bowling as a group?
if you are set on the pool idea, prepare to spend most of your time supervising those hesitant ones. I would agree w/the shirt painting idea, but at a "public" pool that might be hard to pull off.