I understand getting everyone's experiences, but in the end it is only what you and dh think, not your friends and family. Yes, you will have to keep pace with your daughter (but if she naps in a stroller, is easy going) it won't be bad at all to fullfill yours and dh wishes.
There is no right/perfect age or minimum age that you should wait till. It is different for every family. How often you'll take the trip is a big consideration. I think a lot of it depends on what you do at home also. We go to Six Flags, Renassiance Festival, Sea World, Museums, movies, out to eat at sit down restruants, the zoo, the park, playdates, etc all the time anyhow, so Disney wasn't a complete culture shock for our children the first time they went.
The first time I took my kids DS#1 was 3.5 & DS#2 was 7 months. It was a wonderful trip. Everything is so real to them at that age. The baby even had a good time. I don't understand how the stroller & diaper bag & other baby stuff you might need to do is really any different than what you are doing at home, but again, it depends on what you do at home. DS#2 was a bit pre-term, had RSV which led to a lot of secondary problems at 2 weeks so we even had to haul our own pack-n-play, deal with 8 different medications, have regular breathing treatments hooked to a special type of nebulizer, do bleach baths, & a bunch of other junk and I STILL think it was fantastic. We're doing it again this May when the boys will be 4 and 15 months and 5.5 months pregnant! My only regret was not taking DS#1 sooner--we were supposed to, but I was on bed rest with that pregnancy.
The first time my parents went to Disney, I had just turned 2. They left me with grandparents and took my older sisters 7 & 9, mainly due to the huge age gap. My mom felt so guilty since she saw everything I would have enjoyed that they had to go back the next day also (remember, this is when Disney first opened so it was only the MK). According to my mom, everyone had a better day the next day with me there. And she never left a child behind again (I do have a younger brother, 8 years my junior).