Jamaica is another island of a lot of poverty, a seriously rich upper class, and whatever middle class there is not terribly obvious to guests. You see the poverty and you see the wealth.
There are very beautiful places on the island. There are unsafe places. I won't say that I'd never go back, but it wasn't my favorite island. I was there for a week. Our group had a driver who made it his mission to keep us out of trouble. He made it clear to us that we shouldn't go to some areas without him. Hard to interpret how much of that was for our safety and how much was for his pocket...but he was paid by the day and got a generous tip....he did a lovely job and we tended to believe what he told us.
I think that (as many islands), the "touristy" places were fine. And, as above, the panhandlers, etc. can be aggressive, but just tell them "no" and move on--they won't follow you. (or didn't us)
You might want to look up a bit of history of the island....Capsule summary of what we were told is that Jamaica and the Cayman Islands were both under British rule. Jamaica fought for independence, and got both their independence and the government that they then created. There is a lot of unemployment and bureaucracy and many people leave Jamaica to work on Grand Cayman, cruise ships, etc. The Caymans chose to remain under British rule and have thrived with main economic sources or banking, tourism, and medicine with almost full employment. I DO NOT claim that this info is accurate, just that it is what our guide told us.
Edited to add that we spend the week based in Montego Bay.