Funny, we did that very same thing some years back and I'll be perfectly honest. I was disappointed. BY far I could tell the differences in staying at a deluxe vs a moderate.
Although POR is a quaint area there's no comparison to the WL. Just the room alone is set up different. I thought the interior room at POR was similar to a motel set up with the closet area in the bathroom sink area opened. We had to keep the curtains closed because everyone walks right by your room and your window. WE actually had a corner room which was a bit roomier but when we stepped out there was the parking lot, no balcony to relax on. And, if it rained you better have an umbrella cause you'll need it all the way to your room.
We heard the family next to us at POR all different times a day. The connecting door only made it worse.
The main pool area was completely packed although I know WL's can get busy too it just had a whole different feel to it. Boatrights was ok but otherwise the only other choice was the food court.
Probably the biggest drawback is the transportation. You have to bus everywhere. You can take a boat to DTD but it does take a while and sometimes the waits to get on the boat are very long. At least at WL the boat runs frequently to MK. You are near the FW campground and CR which is right on the monorail line so there are lots of choices for dinner.
I know there will be others that will dispute my post but that's just personal preference. IF you do go to POR just don't have any expectations. It won't be the WL because it is a moderate but some people love the moderates. I'm just a deluxe snob (I found that out after our POR stay and haven't stayed at a moderate since).
If you're looking for a more quiet, laid back atmosphere I STRONGLY recommend a place like SSR or OKW. They are probably some of the quietest and relaxing resorts on property. The Swan is also another option.