We are very comfortable driving around the World. I would not have a problem doing this myself.
I would watch the parade somewhere in the first aid / Pizzafari area so that you can head out immediately following the parade. You should be able to be exiting the park around 4:15 at the latest so that gives you half an hour to get to the car and drive to the Poly and still have 10-15 minutes before your 5:00 reservation.