I worry less about the price of fuel at the pump and more about how much food, electricity, water, clothes and so on will go up because of the price increase of fuel at the pump. It is an across the board type of thing. In addition your property taxes will increase and since some medicines, clothing and food are produced as byproducts of oil so they will increase in price...
To answer your question....no i don't worry about it. I work hard and when it is time to go away, we go. It will cost what ever it costs.