I don't know if they are jelous.....we go at least once a year, but I budget at Disney and I budget at home so that we can afford things. The same people who will look at me and wonder how we can afford it seem to be the same folks who buy new cars every few years or get a $70,000 loan to redo the kitchen.
So, it really is about where your priorities are.
We owe on nothing except our house and even our house is 1/3 paid off.
Dawn
That sums it up for us too. We get a lot of the "you're so lucky, we could never afford to go to WDW" comments. First off, it can be as cheap or as expensive as you want it to be. Second, the priorities. Ours are to vacation together as a family at least once a year. We don't have cable, our TV is almost 10 years old and a big behemoth, but hey, it still works!!. My cell phone is four years old and is starting to show it's age. For the longest time DH and I shared one cell phone. He has his own now since when we renewed our contract it was actually cheaper with the extra line than it had been before.

I shop sales for everything that I can. I cook and bake most everything that we eat. We don't drink soda or juice or fancy drinks. Our cars are a 2003 and a 1995. We're not into fancy home furnishings or clothes. So when someone with two brand new cars, a huge mortgage payment (like my SIL who thinks she can't possibly have kids while living in the house they live in now- it's *too small* for them now, with three bedrooms and 2 baths, double garage and decent yard) and who also buys the latest and greatest of everything that comes out (cars,phones, computers,etc....) can't understand why we have the money for WDW and they don't, it really is about priorities.
It's also a priority for us because of DH's work schedule. DH has a part time job from approximately 4:45-7:45 and then goes to his FT job from 8:00- anywhere between 4:30 and 7:30 am, M-F. Obviously the man is exhausted when he comes home. He sleeps most of the time when he is home. Since we pretty much only see him on the weekends, it is nice for us to be able to have a week or two of uninterrupted family time. Oh, I also watch my cousin's two (soon to be three) kids, so even in the mornings and afternoons when we see him for 1/2 an hour, it's not true family time with the other kids there too. Technically they are family but it's not the same, kwim?
The last thing, and I know that there are a lot of people who find fault with this logic, but life is just too short. You never know how much time you will have together so we believe strongly in enjoying the time we have together while we still have it. This is mostly my issue because my Dad died when I was a child. He was diagnosed with cancer when I was 8 and died when I was 11. Those last three years together were spent largely in hospitals with him feeling crappy. When he died he was only 36. I was 11 and my brothers were only 8 and 5. You really just never know what life is going to hand you. I choose to enjoy what we have while we have it.