I've done dinner and breakfast both at least 3 times in the past 3 years. The food is a lot better at breakfast, I especially like the red velvet pancakes! As for the characters:
The breakfast gang (Alice, Mary, Mad Hatter, Tigger, Winnie) has consistently been charming and fun. Out of the three times I did dinner, our character experience (Cinderella, Charming, Tremaine and the Steps) has been inconsistent. Our first dinner had some highly entertaining and fun interactions, our second one was rushed and not that great, and our third one was pretty mixed. I will say that the step sisters have always been funny and interactive, but Cinderella and Prince Charming are both stark hit-or-miss interactions. I should also note that none of the characters are guaranteed at either meal- during our second dinner, we saw Snow White instead of Lady Tremaine.
The strawberry soup, which they serve at both meals, is one of my favorite things to eat on property, so there's always that. If you have someone in your party who has their heart set on meeting someone in the dinner rotation, then by all means book the dinner, but if it's not important to you, I would personally choose breakfast.