My mom took me and my sister one year, leaving my dad and brother at home. A few years later, she took my brother and left my dad, my sister, and me at home. I just think that my dad didn't enjoy Disney like my mom, and, knowing my dad, he probably didn't want to be left alone (so she left my brother home on one trip and me and my sister on another).
I appreciated the time with my mom, and she is much more into going with the flow than my dad is. It is just one less person's desires to worry about! (As an adult, my mom and I have traveled together, and I still look back and appreciate the time that we were able to spend together without the rest of the family.)
From experience, I don't think that it will bother your kids. Sure, they will miss him, but it is some quality time with you. Think of it that way - it is a passion that you have that you can share with them. There must be a passion that your DH has that you don't have, but he can share with the kids.