Rule of thumb for me is to allow 60 minutes to get anywhere on Disney property using Disney transportation. If I wanted to get there by 5pm, I'd leave 3:45-4pm. Since you have a window you want to arrive between, I'd agree that 4-4:15pm would be a good time. You'll likely arrive early, but that's always my preference so I'm not rushing or worrying.
Where will you be eating? The Disney bus drops off at Town Center. If you're dining on the Marketplace side, you could leave later and take rideshare there. The drop off for rideshare is at the Marketplace end. I would allow maybe 20 minutes to call an Uber, have it arrive, and make the commute.