I think shorts are fine as long as they're khaki shorts and not "athletic" shorts.
You see alot of businesspeople eating at the CG, and they're usually in suits or at the very least a polo and khakis.
But tshirts and shorts should be fine as long as they're of taste. I wouldn't recommend a Dirty Dick's Crabhouse tshirt, but a clean, nice but casual outfit should be fine.