We've done it a few times. We always do it in the middle of the trip. We've used Lyft-- and pre-Lyft days, we got a car service. I think Lyft is the most convenient. It doesn't take long to get there by car. We always get the 2 park ticket AND the express pass. Harry Potter is split up into 2 parks, so you will need admission to both. The express pass is expensive, BUT it allows us to do most of both parks in one day...which, in the end is cheaper than staying an extra day, or giving up an additional Disney day. It is also possible to stay on site, and get a free express pass and early entry, etc. but I've never tried that.
Getting there at rope drop, using the express pass and having a 2 park ticket should allow you to get a lot done. Of course, this was before the 10 hour wait for Hagrid's Motorbike. Hopefully that is all sorted out.