For my family it's a matter of our income. When we were first married and had children we just couldn't afford it. Over time DH finished graduate school and got a better paying job and we can now afford it if we opt to stay there. Sometimes, for special trips, we slurge and stay at which ever resort we want, sometimes we don't. We've only stayed at the Poly once, mainly because we just don't feel the need to spend that much on our resort for every trip. The one time we stayed there we had a 40% off military discount, which helped tremendously.
If your heart is set on it, but you just can't afford it, the best thing to do is skip a few vacations and save up for it.