Actually, it's quite simple to get to Fort Wilderness from the Poly.
Just take the monorail from the Poly to Magic Kingdom.
Get off at the Magic Kingdom and take the direct boat to Ft. Wilderness. From there it is a short walk to the restaurants.
If you take a cab to Ft. Wilderness, then the cab will leave the Poly, have to pass the Grand Floridian and drive all the way around the Magic Kingdom parking lot, then come back up World Drive and back through the Magic Kingdom toll booth then turn down the road to Ft. Wilderness. You will be dropped at the front desk of Ft. Wilderness. From there you will then need to walk over to the bus stop and take a Yellow, Orange, or Purple bus to get to the Settlement area. (Yellow is the quickest, then Purple, then Orange). From the Settlement bus stop is a slightly longer walk to the restaurants than from the Marina.
Actually the more direct route from the Poly to FW by car/cab is to make a left out of the main entrance. Go past TTC and make a right at the traffic light (going straight would take you to WL). Just before the MK tolls you will make a left (you have no choice since that is the way the road goes) and this road will bring you to FW. You then need to park/be dropped off at the visitor lot and take the internal FW bus to the settlement depot. Total time for this is about 45 minutes.
It would be about the same to take either the monorail or the boat to the MK and then boat to FW.


