Thanks for all the feedback. It has given me a lot to think about. (Mostly I'm typing this here to help myself think it through and also for anyone who might be considering a similar plan).
We are both in our very early 30s. For this trip, $250 is about 1/6 of what we are hoping to spend ($1500*), so it's not an insignificant amount, although we would certainly prefer to fly. We could save the $250 altogether or put it toward more TS meals. The goal of this trip would be mostly to relax and just enjoy being at Disney World - no need to rush around or try to fit in "everything."
Our flying scenario has us flying out at 1:25 (packing lunch to eat at the airport) - we would have to leave for the airport NLT than 11 am; we figure it would be 5 - 5:30 pm by the time we take DME from the airport to the resort, so we would probably just eat, relax, and go to bed on the first night. We'd stay 4 nights, and fly out on the evening (it's a 6:45 flight) on the 5th day. Total for RT airline tickets is a hair under $500, plus $60 to park at the airport for 5 days, and of course I forgot to add in paying for water once we're through security (let's call that $15). So total cost to fly is around $575.
Driving down we would leave home around 1:30 (eating lunch at home) at about the time the baby would nap, driving as far as we could stand before stopping for dinner/PJs, and then finishing the trip. Realistically, we'd probably do about 4 hours before dinner, then the other 9ish after. We would make reservations for that night so that our room would be ready when we arrived, even though we wouldn't be in it for the whole night (I hope this is even possible!). Our hypothetical plan is to depart WDW sometime in the afternoon of that 5th day, and again drive until dinner, then finish the trip after dinner. Round trip, we figure we would need roughly 6 tanks of gas at about $45/tank, so $270. I rounded down to a cost difference of $250 because you just never know. Wear/tear is a minor factor for us.
So minus sleeping in our room for the first night of our reservation, it works out to almost the same amount of time at Disney. Having our car would be a bonus - the "extra" TS meals we'd do would probably be at resorts, so being able to drive ourselves would be a major plus.
*we've never budgeted for a Disney trip before, but we're hoping to keep costs down for this one: using our APs right before they expire, renting points for an OKW studio to keep our lodging under $500 for 4 nights, and only doing 1 character meal and the rest QS meals. I'm really hoping the price of the airline tickets comes down, but again, I'm really grateful for all the feedback and ideas about driving and the next day. We'd probably rest/sleep, eat a good breakfast, do a park for a couple of hours, come back for naptime, and then maybe do something else in the evening.