Using Disney tranportation, I would say you need to give yourself about an hour.
However (and someone correct me if I'm wrong), I think there's a walkway from the Poly to the Ticket and Transportation Center. If I recall correctly, it's about a 10 minute walk. And you can get a bus at the TTC directly to Fort Wilderness. If all this is still accurate, you probably only need to budget 1/2 hour to 45 minutes.