They will only stay with you as long as you are interested.
My wife and I went to WDW 3 times before our daughter came along and ate some character meals. We kind of enjoyed getting a picture or two with the characters but they moved on very quickly.
In July 07 we were there with our daughter 3 and niece 10 and they spent more time.
I would just get a few pics and enjoy the experience. If you really don't want to deal with them and you are at a buffet, just go up for more food as they are getting to your table. This is like doing the trips to the buffet in reverse because when you are there with kids you have to keep an eye out for the characters so you don't miss them!
We found the food at all the breakfast buffets to be worth the effort even if you are not into the characters.
We liked:
Crystal Palace
Play-N-Dine
Princess Norway Breakfast (Not a buffet, but all you can eat family style with a cold buffet of meats, cheeses, fruits, etc.)
You can't go wrong with any of these! You will be full and won't feel like eating again for most of the day!