Book the trip for just you and the kids. There's no sense in going with someone who will spoil the fun for you.
I'm a single mom and have taken DD by myself twice. Once at 3 and once at 5 years old. I'm planning a trip for the two of us for next August when she will be just shy of 7.
I was a little anxious the first time, but my DD and I had done one day in the MK with my parents when she was 2. I was miserable. All my parents did was complain about long lines (it was early February and the longest line was to meet the princesses which was 30 minutes, everything else was 10 minutes at most) and the cost of everything (but they wouldn't let me pay for anything). I decided that day I was coming back the next year for a multi-day trip, without anyone but DD. We had a blast.