Absolutely no need for your passport at
Castaway Cay. (You aren't going to be late from a shore excursion.) People will argue both sides, but I always carry our passports in a travel bag around my neck when we are on other excursions. If you miss the boat, you'll need your passports to fly home. (Yes, they will enter your cabin and try to find passports to leave with the company agent on the pier; however, I feel better having them with me.)