This question comes up in one form or another about every week and tends to be one of those "hot topics." You can do a search on "pool hopping" to see what I mean. Like most "hot topics" on the DIS you will find people who make a huge deal out of something very minor...
Official WDW policy is that each resort's pools are for the exclusive use of guests staying at that resort. The size and furnishings in the pool area are based on that resort's capacity, not how many additional people might come from other resorts or off property to use the pool. Also, don't forget, part of the reason POR costs more than POP is extra amenities such as the themed pool with a slide - guests staying at a Mod pay more for these amenities.
In reality you will find people can and do have guests visit them at their resorts' pools. Some don't know about the DIS and don't think anything of it. Unless it is a particularly busy time and very warm it will likely not be an issue, except perhaps at SAB and the Poly pool.