I wouldn't. I like babies at WDW. It was by far the easiest place to vacation with infants. My opinion only!!!! Not a judgement on you or your choices!
DH and I did it once when 1st DD was 8 months old. We left her with Grammy and went on our belated honeymoon. We had a terrific time, but missed her unbelieveably. We had booked 7 days for the trip, but ended up going home a day early because I couldn't take it anymore! Now that I have 2 teens and a pre-teen, I'd gladly take a 7-day break!
I have a SIL who frequently leaves her youngest with a relative while she and her hubby take the two older kids away. Granted, he's quite "spirited", and she says he's "hard to handle" and they can't have a good time when he's along . That makes me so incredibly sad. So far, they've left him for a trip to Disney, Myrtle Beach, and just this past weekend for a 4-day skiing trip. He's 6 and I wonder if he looks at those "family" photos and wonders, "where was I???"