Just to be clear: you do have the option of taking the large ferry boat from MK to the TTC, then walking over to the Poly from there. That is a viable option, and is not a bad one. We just find that to be a little crowded and hot compared to other options.
But I was referring earlier to the much smaller boats that go directly from MK to the Poly and GF. You'll probably stop at GF on the way to the Poly, but that's still fewer stops than riding the resort monorail loop all the way around, and you're avoiding the bottlenecks that CR and TTC often seem to be on the resort monorail line. I'd go with these small boats from MK to the Poly, if you want the quickest way for an ADR at the Poly.
Another way to do it would be to just eyeball which mode of transportation: the MK-TTC direct monorail, ferry boat or Poly boat is closest to being ready to board when you walk out of MK, and head for that mode. Any one of those three options will get you there in plenty of time.