I guess you would have it all to yourselves for lunch, but you'd have nothing to eat as Cape doesn't serve lunch, just breakfast and dinner.
While we enjoy the breakfast, we did dinner here last Friday night and I probably wouldn't go back again for dinner as I didn't think it was all that great...or even good. We had ADR's for 7:30, bad time!! It was packed. Lines on each side of the buffet and out of half of what was up there. The salad area was almost all gone as the kid's stuff was slim pickens. It took about a 1/2 hr til most of these things were replenished so we spent more time going up and standing in line than sitting and eating. Kinda tough and frustrating when it's just 1 adult and a child. Won't be going back for dinner. The food was ok, the desserts, well the chocolate thing with the oreo cookie on the bottom made up for the aggravating dinner.
I believe they open at 5:30 so if you're looking for late afternoon, that'll be it. It's probably your best bet as most of the food items should be stocked up. That weekend is the marathon so it could be rather busy. I'd definitely make ADR's if I were you.