Easily walkable, but you can also jump on the HOHO trolley (there are two companies) and hit a lot of the highlights as well as get a history tour. The Southernmost Point marker is only about 1 mile from where you either disembark or are dropped off, depending on your pier. You'll pass the Hemingway House on Whitehead as you head that way. I'd recommend going ahead and getting to the marker first , since that's your most important stop, backtrack to the Hemingway House, then if you insist on going to Sloppy Joe's then head to Capt. Tony's first, since that was the original Sloppy Joe's location and has Hemingway history, then just cross Duval for Sloppy Joe's. Watch for the really cool banyan tree on the left at the Banyan Inn - it goes over the driveway - and the awesome kapok tree outside the courthouse as you walk down Whitehead initially. You can do it all easily in 4.5 hours if you kind of rush the Hemingway House or, instead, only stop for a moment at Capt. Tony's and Sloppy Joe's and spend more time at the house.
Alternately, if you're thirsty for libations and want a more real taste of Key West after the Hemingway House, stop at the Green Parrot on Whitehead. The best dive bar on that side of Duval. My favorite dive bar on the other side is Schooner Wharf Bar at the Bight. You can even stop at the Parrot on the way to Capt. Tony's and Sloppy Joe's, since it's on the way. I recommend the Root Beer Shooter for quick refreshment, or just a cold Green Parrot Ale.