In my experience, debarkation varies quite a bit, depending on how quickly Immigration gets done. I've been off as early as 7:30 and as late as 9:00.
The ship usually unties and pushes off right at 4:45, so they ask that you be back by 4:30, but I've seen people get on right up until the time they pull up the gangway and close the hatch. I've also seen people miss the ship and have to catch a ride out with the harbor pilot launch, but I wouldn't want to count on being able to do this and would shoot for a 4:30 return.