If you choose to stay on the ship on Castaway day, you can eat at the buffet at Parrot Cay or on deck 9 at the fast food outlets. OP's question concerned avoiding buffets. There is not a sit down lunch option normally on Castaway day--just the buffet in Parrot Cay and the deck 9 options.
On our Wonder Cruise in January, Parrot Cay was not a buffet on Castaway Cay day. It was sit-down with wait staff, but ordered off a limited menu. Not sure if that happens all the time.