Well going to Downtown Disney from a park, you do have to go via a resort.
You say you are staying off-site, you do know you have to somehow plan on getting back from whichever parking lot you used after your stay at DTD. I'd strongly recommnd driving yourself to Downtown Disney.
If you absolutely don't want to drive yourself to DTD, and you are parking on site. - might I suggest parking at the TTC lot, then using Disney transport to get to the park of your choice. You'll have monorails to MK and Epcot, and busses to Animal Kingdom and Studios. If there are not busses from the TTC you can walk/monorail to the Poly and use their buses.
The payoff comes in the evening, you can catch a bus from DTD to the Poly then walk to the TTC. The only negative may be that the parking trams will have stopped running, and I don't know if WDW turns off the parking lot lights/tows left over cars at some point. You can solve the parking lot lights/tram issue be returning to the lot at the end of your park visit and moving your car to the closest space you can get to the TTC. Don't know about the towing.
If it were me and I had access to a car, I'd recommend just driving yourself to DTD, their lots are free.
Tips:
For Magic Kingdom - take the monorail to Contemporary, then a bus from the CR to DTD
For Epcot - exit International Gateway, walk to Beach Club or Boardwalk, take bus from there.
For MGM - it gets trickier, if the boat is at hand, ride to the Sawn/Dolphin and catch bus from there, or catch bus to Boardwalk and catch bus from there.
For Animal Kingdom - Go though almost any resort, I don't know if going to Animal Kingdom Lodge then catch DTD would be faster than going to any other resort or not.
Last time I used WDW transport they still ran busses from the TTC to DTD. I susepct they halted that practice to discourage parking freeloaders. I do know the TTC->DTD bus was the most packed bus we had the entire trip, and the only bus driver that inspected resort ID's