Drive time is about 20 minutes.
You are not insane for driving over, but saying you want to go at 9-10am is the wrong idea. Everyone thinks the same as you. They think, "Hey, we'll show up around 10am." What really happens is they run late, and arrive the park at 11am. That's when the "heard" of people arrive at the same time, and they all want to go on the same rides as you.
You need to leave Disney around 07:30am. That means you need to be walking to your car around 7am. Someone will forget something and you'll have to go back.
The plan can work any day of the year, but you have to arrive the parking garage at 8am, otherwise it doesn't work. If you show up at 10am I can guarantee you'll come back here with a bad trip report.