First, I have to fess up and say I've traveled with CC debt. However, that is mainly because I put the trips themselves on the cards. Why? Because I want the points you receive from big ticket items. If I pay them off regularly, then what's the difference between cards and cash? I also have a mortgage, a car loan and student loans that kick my behind.
I would say that one's comfort level does have a lot to do with whether you vacation on debt, but that "comfort" means different things to different people. I have good health insurance, I have never worried about the lights being turned off, I have always saved when I could, I work to pay off my existing debts (though perhaps not as quickly as I would without vacations). Were either I or my DH to lose our job, things would need to be tightened significantly, but we would be okay. Mind you, if we both simultaneously lost our jobs we'd be screwed, but I think most couples would be screwed in that situation. In other words, I don't need to be completely out of debt to be in a situation where my way of living day-to-day is threatened. If I were "struggling", then no, I wouldn't vacation, but having a little debt isn't necessarily struggling.
However, I think the another issue is what sort of importance vacationing has to you. For some, its just a way to relax...a treat for getting all your other work done. For me, my DH and my parents (who passed on their opinions to me), we work so that we can go on vacation. Going away from our real lives is what we live for. It's what we plan for months and years ahead. We don't care about clothes or date nights or keeping up with the Jones'. What we like is looking at our vacation photos and saying, "Hey, remember when we went there? That was awesome." So for me, spending on a vacation is not an excess. It's a life necessity.