Although you aren't supposed to swim at other pools, for a change of scenery, we love to visit other resorts for a snack or drink. That is fun, and sometimes you can sit outside sort of near the pool. It's a really fun way to plan where you may want to stay on future trips!
One year, while I was pregnant (and it was HOT out so I stayed indoors most of the trip and didn't visit the parks) my DH had a conference. We were staying at the SOG, and I walked over to the GF and ate at their counter service place for lunch, then I'd get a soda and relax with a book in the lobby. It was so nice to enjoy the ambiance there, as it wasn't crowded. There were lots of comfy sofas to sit at and relax, enjoying the music and all. You could even do high tea there if you like! Just a thought - also once we got drinks by the Poly's pool, just not in the way of guests there.