As I noted in my first post don't skip the parks entirely but I agree that DTD would be a great way to spend half a day.
What's great about the Poly is the limitless transportation options as you can walk to the Ticket & Transportation Center, which takes you anywhere.
We stayed at POR once and my kids loved the boat to DTD. You could walk to TTC and take a bus to POR and then ride the boat to DTD. Spend half the day there and then catch a bus back to TTC and ride the monorail to MK for your 3 FP+ choices.