We are not cruising experts, and have always done Caribbean Cruises. We do NOT take our passports with us on excursions. Having a copy on our phone? Sure. Actual passports in our hand/bag? No.
Here's my thought... while a passport would come in handy if you miss the ship, personally I think it's more likely to be grabbed by a pickpocket/left (in a bag) somewhere. SUPPOSEDLY (never happened to us, just relating what I've read), the ship will take whatever is in your room safe and have it waiting at the dock if you miss the ship. Again, not testifying that that happens, but it's what I read.
I also aim to be back at the dock no later than 60 minutes before "all aboard", which is usually 30 minutes before departure. So that's a 90 minute cushion.
The other question is what kind of excursion are you taking? One from the cruise line or on your own? If a cruise line one, the ship is supposed to wait for you.
I think there are also some countries that want/require you to have your passport on you. Don't think any of them are in the Caribbean though.
Don't miss the ship!