we did it this summer, wasnt an issue at all, the price diff meant we had little choice tbf, it was alot faste rthan mco and the only issue was dragging kids and suitcases to off site car rental, drive was fine, we did a stop at hampton inn in deerfield (which i would deff recommend clean free beer/wine/snack and free bfast, quiet and nice little pool)
then did 3hr drive up to wdw...roads are well signed we had a tomtom but didnt need it, if there is a massive price difference i would say do it.
i agree the last day is long, we left villa at 10am, drove, stopped for lunch, the car drop off, then airport, flights, we ended up travelling for 24hours which is a long day, but was the difference between travelling or not travelling iykwim
hth
tracy