Since no buses travel between Downtown Disney and any park, probably MK - you can walk or take the monorail to the Contemporary and get a bus from there.
Same from DHS really, except the walk is farther. You'd need to go at least to the Dolphin/Swan; there's also Friendship Boats that would take you from DHS to D/S, Boardwalk, or BC/YC. You can get a bus from any of those resorts to DTD.
Fastest, though, is going to be a taxi.