DH and I have been twice. We're definitely flat-landers, we never had any trouble with the altitude in Santa Fe. If I remember correctly, it was a gradual thing, as we were RVing all over New Mexico. We drove like crazy from the East coast and started sightseeing in Cortez, CO/Mesa Verde (something around 6,000 ft), then on to Alburquerque (around 5,300 ft).
If you fly there, I see how it could be a problem.
The only time I've ever had trouble with altitude while traveling was when I was pregnant and hiking. So hard to walk and breathe at the same time..... this was on Mt. Hood. Had to stop and be dizzy sometimes.