Here is an out-of-the-box idea. Why not take the Express directly to Fort Wilderness?
You’d need to explain the situation to the cast member signing you in at the bus, but I think they’d accommodate... (unless there is some specific rule for safety/insurance/liability that says they can only drop you at your reservation).
Call the DME phone number and see if it’s allowed.
I got off at the ‘wrong’ hotel on purpose because I needed to get to RunDisney Marathon Registration before it closed. So, I jumped off the bus as soon as I was on Disney property to take the random hotel’s shuttle to the ESPN Sports Area. That’s slightly different because I was still on the ‘right’ bus whereas you’d need to be on a different route entirely.