Have only stayed at POR but would recommend it. In my opinion, POR is much more of a "get away from it all" type of resort whereas POFQ, I've heard, is a lot more "happening". The buildings are more spreadout, but I think that makes you feel like you have fewer neighbors. No sit down restaurant at POFQ and Boatwright's at POR is great. Also, when we were at POR, because they share a boat to DTD, the POFQ people were sometimes passed up by boats that filled up at POR (strict limits based on lifevests).
Whichever you stay at, take an evening boat ride to DTD. You will pass the old Treehouse villas (now defunct) and they are an interesting site at night all deserted like that.