Coming from the Poly, no, it'd be much more efficient to go straight to DHS, as PP have mentioned. You would be dropped off at the front of EPCOT, have to walk to the back of the park, through the IG, through the EPCOT resort area, past Swalphin, and up to the front of DHS. We stayed at the Dolphin in October and regularly walked to both parks - from Dolphin to DHS was usually a 15 minute or so walk. That was without the 5-10 minute walk from the International Gateway to Dolphin. Robo will probably be along shortly with a map of the area.
Also, if you don't have Park Hoppers, you would use up an entry with this plan - you would have to use your ticket to get into EPCOT in order to get to that walkway.
If you are on a time crunch, you'd be better off calling a taxi or Uber, if you are set on not taking a bus straight to DHS from the Poly.