I have a friend who takes each of her kids w/ her husband to WDW when they turn 5, and she leaves the others at home. It is such a special time for the child who, if you have more than 2 kids, probably doesn't get lots of one on one time.
I have another friend (4 kids) who has been many times over the years and leaves behind whichever child is the "baby", which includes 3 of her children at one time or another, w/ grandparents. They still go regularly as a family, and there's been no resentment whatsoever there.
Meanwhile, only my oldest has been to Disney back when she was less than 2 and we only had one child (now we have 4), and we are planning our first family trip back in 8 years. If I were all of a sudden to take my younger child to WDW there would be he** to pay, but if we, as a family, went often, it wouldn't be that big of a deal... does that make sense?
I guess it also depends on how much emphasis your family puts on Disney. If Disney is the "end all be all," I wouldn't do it, but if it - to your kids - is just another place to go, then it wouldn't be that significant. IMHO.