I'm probably one of the people you're referring to. To explain - millions of Jews died in the Holocaust. Millions of people in the current unrest in the Middle East. Some used it to justify slavery. A partial basis for segregation and apartheid in South Africa. Partial basis for the Northern Ireland conflict. The KKK. Then there are more current nonviolent political issues, such as opposing same-sex marriage on a religious basis. Bandying about the idea of a Muslim registry, even if it was never really seriously considered, is cause for alarm.
I realize that there are many more, and larger, secular sources of conflict - territory, resources, power, etc. I don't want to lay all of the world's ills on religion; clearly that's not the case. However, speaking broadly, I believe religion can separate outward just as much as it unites inward, having serious consequences. So, yes, I have some hostility and am actually surprised that more don't.
I think the presence or absence of hostility very much depends on your experience of religion.
Giving myself as an example: I was raised Quaker. I attended a convent school full of the kind of nuns who played guitar and told me that "God loves ALL His children! Even you!" and never asked me to convert. I moved to Canada and ended up in first Catholic school, then a Protestant private school. Neither ever asked me to become their kind of Christian. I was once pushed into a broom closet with a Catholic priest and I said, "I'm sorry, Father, I'm not Catholic." His reply, "Oh, that's fine. As long as we're here, is there anything you'd like to talk about?"
I hung out with Wiccans. I used Christian homeschool materials to teach my kids.
Later, I became a member of a Unitarian congregation (because my husband couldn't cope with the Quaker commitment to pacifism). At the time I joined, our pastor was an avowed Atheist. We've been through a few pastors since then - the current one is Christian. There's a rainbow flag hanging over our church's front door, and a plaque recognizing that the land our church is built on once belonged to the local Native tribe. We have a Jewish congregation using our building on Saturdays and sometimes the Wiccans rent out our space for rituals.
For me, I have very positive associations with religion. When I read about the grief people cause each other in the name of religion it causes me some cognitive dissonance, because it is so very alien to my experience.
By way of analogy: Imagine spending your whole life around dogs, loving them and enjoying what they bring to your world. Then imagine someone tells you about all the people who've been injured or killed by dogs, and says they hate dogs and can't understand why anyone would want to bring one into their home. You still love your dogs, and you still feel positively towards dogs in general, but you can't dismiss the truth of what that person is saying either. You feel bad for them.
So maybe, if you don't think better of it in time, you might argue with them. Point out all the wonderful dogs in the world. Dogs who've saved lives. You might tell them they just need to get to know dogs better. Heck, you might even give them a puppy as a present. It would be a very ill-advised, unwelcome present, but one which comes from both a place of caring and that natural human desire to make others see the world the way we do.