If you don't take them, you may do what we did...take them 3 months later!
Of course, that was our honeymoon and the trip 3 months later was an awesome deal my DH couldn't resist. But I understand sometimes parents just need adult time. We had a great trip on our Disneymoon and I didn't feel guilty because it was the only trip I ever had without them.
I think the OP is planning to take her two oldest, and leave the two youngest at home with the grandparents.
I have to agree with the previous posters, I took my daughters for the first time when they were 5, 3 and 1 and we had the MOST wonderful vacation! I think that you would be thinking about them and missing them terribly the whole trip!! Good luck with whatever you decide.
I couldnt imagine going on a vacation without my 3 kids. Anything we do as a family we want to share...especially the magic of Disney. I can see why OP might feel a bit guilty.
Your kids might not remember going but you would remember taking your kids.