Actually Disney's written policies are that resort guests, not resort guests and their guests, may use the pools. Keep in mind that during busier times, they do check rooms ID's, and swimmers who are not a guest of that hotel will be asked to leave.
Of course you could all swim at a waterpark.
Anne