It really depends. I am generally a little cold in most of the table service restaurants if I don't have at least a long-sleeved t-shirt or light sweatshirt with me. There are some that are downright cold though. Hoop Dee Doo for instance, the first time I went, the servers brought tablecloths for us to wrap up in we were shivering so much. Same thing at California Grill. So for me, I always bring something with long sleeves even in the middle of summer. The AC feels good for the first couple of minutes after being in the heat, but them it's a little too much in my opinion.
I'd recommend bringing something just in case rather than be caught without it and want a little something to keep warm with.