I drove from AL with my kids when DD was 4.5 years and DS was 16 months old. I was nervous, but the only thing I found difficult -- and it was VERY difficult -- was having a counter-service meal during peak lunch hours. (Imagine juggling a squirmy toddler and a loaded tray, not being able to find a table, and then eating with your tray on top of the garbage can because it's the only flat surface you can find!)
But that was the ONLY bad experience we had, and we ended up with 4 days worth of very special memories.
Then again, my next trip will just be with DD, so that we can make her 5th birthday extra special, doing all the big-girl stuff she wants to do without needing to take her little brother into consideration.
I felt a little guilty about not taking DS, but when he injured himself and needed 10 stitches on our family WDW trip last month -- and DH and I were both in the room at the time -- I realized he is at too active a stage to bring without another adult (or 2!) present.
If your son is not too self-destructive for a 3-year-old, I think the 3 of you will have a great time. If he seems to attract injuries or is prone to wandering off, however, I think there is no harm done if he sits this trip out so you can focus on your birthday girl. You might want to consider making it a family tradition and take your son on a Mommy-and-Me trip for his birthday at a later date.
Good luck with your decision!
