We stayed at the Poly and then headed to the Port about 2 years ago. Our bus left at 11:00 and the ride only takes an hour more or less. I am not sure if the bus times are still the same, though.
They serve the buffet lunch until about 2:30 or 3:00. They serve in two locations, Parrot Cay (Deck 3) and up on Deck 9 in or Beach Blanket Buffet. They stop serving at different times.
MJ