If I were you, I'd stick it out. Cut back everywhere else you can and keep that property. It sounds as if you have enough time to earn more money, build equity on the property and you love where you are.
Our situation was completely opposite of yours.
We decided years ago to stay in our small (ish) home on a 15 year loan. A house I've hated since the day we moved in.

We've paid extra each year and it's almost paid off. I don't think mortgage, taxes and insurance was even 10% of our income at the time. We started looking at homes twice the cost of ours and were so tempted. It was really our kids who talked us into staying. They loved the neighborhood, school and friends. So we stayed.
Two years ago, dh got brain cancer. We are so glad we didn't move on up when we had the chance. We had to use a good chunk of our savings and investments on medical bills and travel expenses for his treatment. We were able to leave our mortgage and 401k alone. So now although he took a very early retirement and things are tight, he is in remission and things are looking bright for our future. If we were living with a higher mortgage, higher taxes, etc. we'd be so much worse off right now. We'd probably be in an apartment.
Good luck with all this, but I agree with your husband!