If you are concerned about character interaction or food freshness - no need to worry; that will be consistent through the evening. I think there are more young children if you eat early.
We generally eat late -- around 8:30 and found that as the restaurant empties the character interaction actually intensifies because there are less tables that still have guests. We had the best time ever last January, closing down the place with Lady Tremain and the stepsisters.