During the week I've found that lunch times are not as busy as in the evenings, and found that you get more character interaction at lunch time than at dinner.
At weekends it tends to be busy at both lunch and dinner time.
We went at lunch time in the summer holidays and it was very quiet without much atmosphere. Would like to try evening now as I think it would have a better atmosphere and probably more characters by the sound of it