Normally I am not one to say "stay home" with an infant. In fact, we would have gone to WDW with my 3rd at 2 weeks old last year, but I ended up with an emergency c-section, so my husband didn't think I'd want to make the drive and instead he called my mom to stay for a week, his mom for a week, and his sister for a week. Well, there was a blizzard and turns out his sister couldn't get a flight in, so instead, the 5 of us were snowed in at home, it was too late to get to WDW (as my husband had to be back at work the following week, so with a 2 day drive roundtrip, wouldn't have happened), and we were all stir-crazy and annoyed. So we drove to my mom in NJ instead!
Anyway, here's my point. My cousin just had her 3rd baby on March 2nd. He was a full term induction, sent home with mom on the 4th, healthy and happy. On the 6th, my cousin noticed that the baby was starting to breathe funny, like he was out of breath. They took him to urgent care where she was told "Your baby is too sick to be treated here, we're taking him via ambulance to a medical center with a specialized PICU 1.5 hours away." Once they were there, she found out her baby was born with 2 congenital heart defects and would need immediate surgery. BUT, since this was undiagnosed for days (and not visible on the prenatal ultrasounds), his lower organs had been deprived of oxygen and he was in liver and kidney failure, and his intestines has slowed down as well. For 48 hours they didn't know if he'd survive at all. He was on a ventilator, more medications than you could imagine, was under sedation to keep him calm and his heartrate low, couldn't be held or touched by his parents, etc. 10 days after he was brought to the hospital, he was healthy enough for surgery and underwent a 10 hour open heart surgery which, thankfully, repaired both of his defects. That was over a week ago and he is still in the PICU, just got off the ventilator yesterday, heart is working perfectly, but he's still on tons of meds and has a nasal cannula in. He is 3 weeks old today.
Based on his sad, sad little story, I would say plan to stay home and let your husband take your older kids. You never know what the baby might be born with, what issues could develop after birth. My 3rd had severe reflux and at 3 weeks old spent her days and nights SCREAMING cause we had no idea she was in such pain. It wasn't until her 1 month check up that we diagnosed the reflux, and not until 6 weeks old that we got the med dose correct, and determined dairy and nut allergies, that the screaming died down a bit. And my husband is a physician yet even he had no idea what to do!
I am sure dad can handle your older ones on his own, they'll have fun, and a break from the baby stuff, and you'll get time home to bond with the little one without the others demanding your time and attention.
In fact, that was our plan after my c-section, my husband was going to drive to WDW with the older kids while I stayed with his sister at our house. But with the snow he couldn't leave when he wanted to, then when his sister couldn't get on a flight he felt badly leaving me home alone at a couple of weeks post-op, so we all stayed home. The kids never knew of our plan so they weren't disappointed.