Fantasia Gardens Minigolf is over near the Dolphin Resort, there are no buses that go directly to the Minigolf itself.
You can walk to it, here is the info from Deb Wills site:
ON FOOT: From Disney's BoardWalk Resort, take the bus walk to the boardwalk. Turn left and continue to the end between Atlantic Dance and Jelly Rolls Piano Bar. Cross the bridge toward the Walt Disney World Swan and Dolphin and continue straight ahead to the street. Fantasia is on the left and the tennis courts are on the right. (If the big tent is set up, guests may walk through the tent.)