I think I'm heading towards the 'realisation' that the OP had too...
I have been so lucky, and have managed 5 trips to the World since 2001. Despite financial setbacks (divorce, and then meeting my new husband who has two kids we pay for half of everything for).
We went as a family of 4 in 2010, partly funded by having more on the mortgage than we actually needed to move house, and partly funded by the credit card.
We went last year for our honeymoon, just the 2 of us. Completely funded by the credit card in the end, as my financial planning was thrown into a spin by the wedding costing more than we thought it would, and a work bonus we were expecting not being nearly as much as promised.
So we're now paying off the credit card, going to Whitby for a week this summer (which is fine, we all love it there too!), and hoping to go back to the World in 2014, as a family of 4, but the kids will be 12 and almost 17 by then, so 4 adults!! That really ramps up the costs, as others of you have said.
But I'm slowly realising that we just won't have enough money saved up to go in 2014, as it's going to take us a while to clear the credit card. I vowed I wouldn't go down the credit card route again (I feel a bit wobbly doing it that way), I said we'd save up then go. So it looks like it might be 2015 unfortunately, which will disappoint my step-kids I think, but I think that's the choice I have to make...
Unless something magical happens before then, you never know!

The couple of work bonuses we had have been unexpected, so maybe something nice will happen which will allow us to go in 2014.
We too, love to stay on site, and fly direct. And I'm happy to only go every 2 or 3 or 4 years to achieve this. Although as the kids get older, the indirect flights look a bit more attractive...
I'm VERY grateful that I've had the chance to go so many times, I know there are many people who will never be so lucky, and I'm lucky to have a happy, healthy home life in a reasonably nice house (although it could probably do with a kitchen!). It still smarts that everyone's earnings just haven't kept up with the cost of living, and we are all feeling the pinch, and working harder to continue doing the things we like to do.

to everyone out there!!