Being a DH myself I agree with this point. Saving for a DW trip can be hard with all the things families do these days. I would be soooo jealous if my DW and kids went w/o me but that would never happen b/c I love it too much.
However I can see where some times you might want to skip it for a year or two. That makes going again even more fun. You need to really dig deeper and see if he really won't mind you going or if he is just saying this to make you happy.
I could fill in as a DH if you need one, lol
Hope this works out for you.