When is your trip?? If it is in the near future, you can walk to the TTC from the Poly, then take a bus over to Wilderness Lodge. (This is because there is current construction at the MK bus stop that requires the Wilderness Lodge bus to run to the TTC.)
From the Wilderness Lodge to Ft. Wilderness you can take a boat over, or take the walking path over (about 20 minutes).
To get back to the Poly, if the Magic Kingdom is still open, you can take a boat to Magic Kingdom or the Contemporary, then take the monorail to the Poly.
OR take a bus to TTC from the Settlement bus stop, then walk to the Poly.