It's the city. We actually sold our original house here, which was HORRIBLE, and bought a different one almost 2 years ago, knowing we weren't moving any time soon. We love our new house.
As for the city, it just isn't a good fit for us. It's a tourism area and a retirement area. For those who live here, some "locals" make a point of making sure the rest of us know that they would rather us not be here. I actually read a comment in today's paper about how "outsiders" try to come in and change things. Last time I checked, I pay taxes too and I've lived here for 7 years. It is very clique-ish and the cost of living is one of the highest in the southeast (though compared to many other places, especially the northeast, it is low). The really bad part about the cost of living, though, is that it doesn't correspond with the wages. Housing is particularly high.
DH and I are also hot weather/humidity people. Aside from the recent heat wave, it just doesn't get hot enough here for us.