Been to Nassau a few times now in the last few years, most recently last year. As I mentioned earlier, we had our 5 month old. We have never, ever felt unsafe there at all. The Bahamian women definitely said 'Hello' to my baby, commented on how cute he was, fawned over him, but never was anyone aggressive or threatening to anyone in our party.
In the Straw Market, they are trying to sell. If you are worried about being 'harassed', I'd avoid it. I loved it. But, if being approached by anyone worries you, don't go there. You will be talked to, but I find them friendly!
When we go again, in less than a month, we will be probably strolling the streets, may take a cab to the Fish Fry, and if we are in a beach mood, we will walk next door to the port to the Hilton where you can use their beach with a day pass.
Last of all, we haven't been to Castaway Cay with child... but we have been without child and also to the beach with him. We plan on bringing a floaty toy for him (one that is inflatable), lots of sunscreen and a head to toe outfit for sun protection, lots of drinks and a good attitude. We tried to bring a 50 SPF beach umbrella with us, but it got lost by the airline. We did find a protective sun net made by Jolly Jumper that is pretty cool to bring and we've used it on a few trips now.
Traveling with a baby just requires patience and planning more than anything. Run through your day and pack all of that along with anything you'll need for the sun
You'll have a great time!