A bit of info is missing here. the answer REALLY depends on your starting location, and where you want to go AFTER your mini-golf.
You can't take a bus from any ol' resort to the Swan. The only resorts that share easy transportation to the Swan are the Dolphin, the BC, YC, and Boardwalk. If you are at one of those resorts, you can take a the boat to the Swan. Then again, if you are one of those resorts, you could also walk over. Generally walking is fastest. The boat in question ALSO makes stops at HS and Epcot.
If you are at MK, AK, DTD, or Typhoon Lagoon then you would take a bus to the Swan. (If you are at Blizzard beach, you'd simply be better going to the mini golf there).
If you are coming from any other resort, you would need to do a two step transportation transfer. You need to first take transportation to a park - THEN transfer to a Swan bus.
If you are at CR, it would be easy to simply walk tot he MK bus stop.
Otherwise, if you at all think time=money, then likely your best value would be to hop in a taxi.
Where are you staying? Clearly, if you area t an Epcot area deluxe - getting to the Swan would be easiest. CR to MK to the Swan is also pretty easy.
If you are coming form almost any other resort, Winter Gardens (via BB bus) or taxi - is likely to be your easiest option.
Transportation AFTER your game is another matter...Going from Fantasia say - to one of the major parks is pretty easy- use transportation from the Swan. Not sure how late buses run from Blizzard Beach. I think most days BB closes at 5pm, so if you plan to play later than 5pm, I'd double check transportation is available. (that may help explain the 4pm start time limit.) Again, in most cases, getting back to a resort from Fantasia is likely to be a two step process where you first walk to the Swan, then go to a park, then take another bus back to your hotel. Or....hop in a taxi.
FYI, we've never bothered with WDW mini golf. Transportation is a big part of why they've added it as a 'free' bonus.