It's unfortunate that downtown (the port area) isn't cleaner because so many people get off the ship, walk around town and that's their only experience, so their impression is that Nassau is run down and dirty looking, plus you have the peddlers outside the terminal building which can be annoying but we just always keep walking and say no thank-you (although we've never had one continue to badger us for a tour once we say no).
We've been to Nassau three times, and other than a little shopping we always get out of downtown, twice to Atlantis and the other time we did a snorkeling excursion on the other side of the island and we had a great experience all three times. One of those times was a 2-night/3-day mini vacation and during that trip I discovered how friendly the locals are, more so than other Caribbean islands we've been to. Everyone in the Paradise Island area (which is where we stayed) greeted us with a hello and a smile when we passed them, the store and restaurant employees we dealt with were friendly and our taxi driver to/from the airport was great, and like the others, very friendly. We just found it interesting the difference between being in Nassau for just a limited number of hours during a cruise stop vs spending a few days there.