Well, we go every year and my kids have had their first trips at about 7 months. My favorite age to take them was 3 and 4 years old. They still believe in the magicas in it is *real*, not magic. That is amazing and its impossible to explain to those who havent done it. There is just something special about Walt Disney World and kids that age. Any age is good, dont get me wrong, but those trips are different. Still fantastic, but different. Things really change by the time they are seven or eight.
However, there are a lot of things to consider. WHY does your DH think it is too young? Because your 4yo wont remember it? Maybe, maybe not. On our last trip our youngest (who turned three right after we got home), was able to direct us through the park. We followed all his that ways until we heard There it is! Buzz Lightyear. He knew where he was going, from memory. As in, he remembered. I remember select items from when I was there at two years old. However, chances are pretty high they wont remember all of it the way some people want to at that age. But, that isnt a guarantee at any age. I dont remember many/most of the things that happened on trips when I was fourteen years old. A trip at this age should be about your memories and your childs experiences.
Does your DH think your 4yo is too young/too short to experience much? Well, very little has height restrictions. By rule, there is way more that can be done by anyone than cant. WDW is way more than rides anyhow. There are tons of shows. There are the parades and the fireworks. Is he afraid it will frighten your 4yo? Maybe, maybe not. My kids have all been on Dinosaur at age 3 and LOVE it. My kids have never been afraid of Its Tough to Be a Bug. It even got my youngest over his bug phobia. My oldest and youngest have loved the characters from the beginning. My middle one was terrified of the characters for the first few days at 15 months, but then warmed up to the costumed characters. By the next trip he liked the face characters. Something about the characters talking didnt appeal to himthe Mad Hatter told us it happens a lot. Go figure. Oh, and my youngest who loved the Disney characters, was afraid of Santa and the Easter Bunny and any other costumed characters at home. Disney was just different.
Does he not want to deal with a stroller? Well, that may be a point, but it isnt anywhere near as bad as some dissers make it out to be. Im actually going to miss the stroller; it keeps me from being a pack mule. And really, one more year isnt going to make a huge difference on this point.
Is it a once in a lifetime trip? Well, maybe a point. But will you have other children? Are you going to punish the oldest because of the youngest? What if a surprise baby were to happen? Besides, nothing in life is guaranteed. I have a friend who thought I was nuts for taking small children. They were waiting until their youngest was 7 or 8 and their oldest about 11. Theyd really get the most of it then. Well, until her step daughter had some type of school thing there. They took the kids at 2 and 6. She came back and said she got it now. It was an amazing trip and she was floored at how easy it was and how much fun they *all* had. A year later her DD was diagnosed with cancer at age three and lost her battle three years later. At the funeral her DS was talking about how that was a great family memory. She thanked me for that because I kept encouraging her to take the kids. She almost didnt go and just let her DH go see her step daughter alone or with their son as a father-son trip. Sorry to be depressing, but it is a good memory for them.