Mainly thinking about on the way back, when your still kind of wet. Even if you put dry clothes over your wet suit. It's usually nice to get to your room to change in to dry clothes as soon as you can.
Regardless the walk alone just to swim is a bit much, even if the pools are nicer. Not quite the same as a quick elevator ride to the pool.