We just got back from Wilderness Lodge and took the boat several times to and from Magic Kingdom. The boat trip took maybe 10 minutes max. We didn't have to stop at Ft Wilderness though, maybe they only do that at slower times.
You can also take the bus, it only takes about 5 minutes or so. Like any WDW transportation=, a lot of it is a matter of good or bad timing. This trip it seems we had great timing, we would no more than get to a bus/boat stop and a bus was either there or one pulled up within a few minutes. Other trips it seems like no matter what we would always get to the bus stop right as a bus was pulling out or a few minutes after it pulled out.