First of all, you can't go wrong with either.
We used to always do the late breakfast there and it worked out fine.
The last trip we already had three other breakfasts (play-n-dine, akershus and Tusker) we wanted to do, so I gave an early lunch a try. I was hesitant at first because I don't like to fix things that are not broken, but I'm glad we did.
First of all, the food is just as good, better actually, than the breakfast. It was probably our best one of the trip!
Secondly, get there at park opening and you can get some serious attractions in before your lunch. That little bit of extra time makes a difference!
I guess there are some other breakfast options that are close to the quality of CP. Akershus for example does not appeal to some kids at meals other than breakfast. Unless things changed, the characters were only at the Tusker b'fast. The lunch and dinner at play-n-dine was getting bad reviews, but we already knew their breakfast was good! This was the biggest factor why we went with the lunch. I'm telling you, it wasn't a sacrifice at all!
The CP lunch is a "must do" for us forever!
Hope this helps!