I have never eaten at Charley's but I know the Yachtman's has super food! We have eaten there every trip since they opened. Personally, I don't like to fight the traffic to go to off site restaurants.
Im another one who doesnt go offsite, but loved YSH.
As for atmosphere we saw all sorts of attire on people there and kids of all ages. However we were seated in our own little corner and it seemed very secluded there.
Been to both and prefer Charley's over YSH. We have dinner at Charley's every trip and have always had an excellent meal. I do like YSH caesar salad better, but Charley's steaks cannot be beat! They also have wonderful side dishes and huge portions.