It's not quite that bad - you'd have to cross World Drive without a stoplight, but that's the only real challenge.
The key is to go out the "back way" of the Lodge, through the CM parking lot. This route would take you to the back door of the TTC. Some of the internet satellite view sites can make it fairly easy to see. It's one of those things that I want to do some time just for the experience.