The walk down Whitehead back to the port is nice. You'll walk past the Hemingway House, the lighthouse, the Audubon House, a *huge* Kapok tree in front of the Courthouse, one of my favorite bars in the world called The Green Parrot, Garbo's Grill (which is actually on Caroline between Duval and Whitehead), Mile Marker 0 (which, to me, is more emblematic of Key West because it's truly MM 0 as opposed to the "Southernmost Point" not really being the southernmost point in the U.S.) and many other things. I like Whitehead much better than Duval but not as much as Simonton.
Be prepared to wait in line at the Southernmost Point depending on how quickly you get there after disembarking, what time of year it is and the port load at the time.