And some of the resorts check the resort IDs of those who are using the pool. It'd be kind of embarrassing to be asked to leave, I think. I've seen some pretty red-faced people packing their stuff at the WL pool after a guard told them to leave.
You're welcome to visit other resorts and shop, dine, rent boats, walk around, etc. Disney encourages this. But pools are only for the use of resort guests. The pools are built to accommodate those who stay at that resort, and are simply not big enough to handle a whole bunch of extra people.
Edited to add: pool hopping is permitted in certain circumstances:
1. If you're staying at one of the All-Star resorts, you may use any of the pools at any of the All-Star resorts.
2. If you're staying at POR, you can use the pool at POFQ. If you're staying at POFQ, you can use the pool at POR.
3. If you're a DVC member,staying on points, you can use any resort pool except SAB at YC/BC, and the pool at AK.