Seriously, the Op's DH and I are the only people who think it is wildly inappropriate for an adult (in this case a grown man) of ANY disposition to be hanging out with young kids?????
#1, it is not a child's job on any level and in any way to take care of, support or help an adult. I don't give a hoot what this person's sob story is... and why on earth would anyone believe it as truth anyway? This isn't a friend's son or someone with ties to the family. This is a grown man with a story that may or may not be true.... and I don't think it is true. Why on earth take this person at his word????
#2, child molesters look to kids precisely because they have been rejected by their peers. In my world him being a loner is a cause for another red flag bells and whistles, not pity.
#3, the father doesn't like it. Why on earth shouldn't the father's instincts be trusted? Most men rarely weigh in on this sort of thing, so when they do I think their words should be heeded. If my DH ever said, "I don't want ---- around the kids", that would be the end of the conversation. I'm a girl so I get how we do things and girl double speak, DH is a guy so he gets the guy code of behavior and guy double speak... I would never ever blow off my DH's opinion on other men's behavior. Why on earth dismiss the kids' father? In favor of a stranger no less, what is up with that?
#4, many child molesters groom kids for years and years before they pounce. Most are not derelicts that grab someone from out of the woods. Most are adults given access to children where the kids feel safe and what is more reassuring than the kids' parents giving it a nod? Ever notice how they tend to be teachers, scout leaders, coaches, boyfriends or girlfriends, step parents, pastors, friends' parents? The one thing all these categories have in common is that other adults sanction the contact with kids which allows them to get in under the radar. All they need is an opening, they'll do the rest.
#5, it's just flat out weird for an adult to spend free time with kids. Look around you for heaven's sake, and pay attention to the men in your world. Men are not women, they tend to only be interested in their own kids, most can barely stand when their kids' friends are around stirring up things. They do not look to put themselves in situations where kids will cause a stir, they skip out on birthday parties and the PTA. It's just not normal guy behavior to go out of their way to spend so much time around children. Your DH knows it and even some of the kids who question this guy intuitively know it. I think even you know it on some level because you are posting here to get feedback. Don't wait to have your gut instincts validated by other people, if you sense something isn't quite right then trust that... and go get the book "Protecting the Gift" by Gavin DeBecker who was on Oprah years ago. The author is a child abuser profiler who mentions different strategies used by them and how to recognize them... this guy you are talking about fits the mold to a "T". Oh, and don't make excuses because you like a person. We all do it, we make exceptions. These exceptions are how these creeps get to kids.
On a softer note, I do get it, no-one wants to make anyone else feel. It is human nature to nurture others but the truth is this need to make others feel better is easy to manipulate. You are talking about kids and an unpleasant truth is there are people they need protection from. This guy has bad news written all over him and I would run him off faster than lightening if that EVER happened around here. In fact, I warn my kids all the time about adults or older kids who try to be their friends. Kids belong with kids and adults belong with adults, no exceptions.
I hope you're not too late already, a great deal can happen in 3 years... these kids trust this person now and he is entrenched in their lives. Good luck