I would pick whatever fits your schedule better. You can also look at the menu and see what food more appeals to you and everyone going with you.
My personal opinion would be to pick breakfast. A lot of breakfast buffets are somewhat the same and offer similar choices (I will add that there are a lot of better breakfast buffets out there like Crystal Palace and Tusker House).
We did CM brunch once, and once was enough. There's too many better places for that in Disney. Our last brunch was at Boma and it was great!