Only done the late breakfast, but I can give you my experiences on that:
We had a 10:25 ADR for breakfast at CP, which we were told was one of the last seatings for breakfast. The place was very crowded and hectic at that time, but not overly so. I thought our service was just average (i.e. took a long time to get our drinks, refills, and check - ended up just paying cash so we could go). BUT the food was excellent! Everything was very fresh and I just love breakfast food. I usually hate WDW coffee, but for some reason I enjoyed it here. The characters were in abundance and came around several times. Tigger even snuggled up at our table and surprised us. I enjoyed the late breakfast time because it really filled us up until dinner, which saves money. We shared some breakfast type bars in our hotel, and then got to MK at opening. So at 10:30 we enjoyed the break.
Not sure about lunch, but I know towards the end of our meal, they took down one side of the buffet and started setting up for lunch. But there were still plenty of breakfast items. The place was definitely cleared out when we left. I prefer breakfast foods so that's my vote, but if you want a quieter experience, an early lunch might be better.