Coming from MK and going out through the turnstiles,
Walk straight toward 7 Seas Lagoon down the middle from the gates (Monorail station on your right, bus stops WAY on your left). The boat dock in front of you will have the Fort Wilderness queue on the right side (left side goes to other resorts). The boats run every 20-30 minutes so to be safe, be at the dock an hour before your brunch time (in case one just pulled away before you walk up).
I enjoy the brunch at TE, Crissy. Having breakfast and dinner food on the same buffet is AWESOME.
Bama Ed