You only go through the Castle to get to Fantasyland, not the other lands.
At the end of Main Street USA, before you get to the castle, is the "hub". The hub has pathways that lead to Tomorrowland, Adventureland, Frontierland, and through the castle to Fantasyland.
It's 1- 2 minutes from the hub to the different lands.