Not wrong at all! You need some time to refresh and rejuvenate....can't take care of the family without taking care of yourself first! Enjoy some magic and some relaxation! This trip can be about YOU and the next one can be for the whole family.....you will enjoy both but they will be VERY different trips!
I just went through this too, I'm a single mom and my mom offered to watch my son for me so I could get away by myself. I felt guilty at the thought of leaving my son and going without him, but I got over it, lol. I think it's really good to have some time for ourselves to relax and have some fun. I bet it would be a good bonding time for your husband and kids. I know my son is looking forward to having a sleepover with his grandparents (although he doesn't know I'm going to WDW, he thinks I'm just going on a business trip). Enjoy!!