So where, exactly, would your husband's co-workers recommend you go on vacation, if not to WDW? I can't think of a BETTER place to take young children for a fantastic time!
On our first ever trip to WDW, DD was 17 months and DS was almost 4. We were a little nervous about DD - she was still in diapers, didn't do loud noises, was a picky eater, and was generally a stubborn, determined child.

Even she had a fabulous time! Truly, it was a magical trip for all of us. I completely agree with the pps who talk about "the magic." Many 8 and 10 year olds don't get it. To see my son sit and talk a princess's ear off, hug characters, and get completely excited about the little things was priceless. You don't get that with big kids. At least not all big kids.
Taking younger kids does require a heightened sense of planning and patience, but as long as you're willing to jump through those hoops it will be more than worth it.
My best piece of advice when it comes to anything Disney - do what feels right for you. Do your homework, arm yourself with all the info you can, listen to advice, and then do what you want. And don't feel guilty about it. When you come back from a wonderful trip, those co-workers will all probably be jealous.
Have a MAGICAL time in Disney!!!