Castle Inn just had a major remodel and the rooms are very, very nice. Beds are comfortable and the overall experience was great. The walk is maybe five minutes longer, definitely not something I would consider in my decision making process.
The Desert Inn's only redeeming quality, in my opinion, is it's close location. The rooms are old, walls are thin, slow elevators, etc. The place is aging and is in dire need of an update.
I would stay at either again, however. But given equal price, I would go with Castle Inn.
Good luck!