OK, first of all, I in NO WAY think/suspect my DH is a pedophile. Let's get that clear. If I did, he would be my ex-H in a heartbeat.
And it's not that he can't/won't spend time with any of the kids in the family- quite to the contrary. It's either that there are more than 1 kid with him (we have 10 nieces/nephews), or I go if there is only 1. Too many people close to us (not just men) have been falsly accused and it's not a risk either of us are willing to take.
About half the kids come from 'broken families', so there is no way I am giving my ex- SIL and/or ex-BIL leverage to get full custody of the kids by claming anything. Because that happens too- kids will say what they think their parent wants to hear, including that Uncle Phillybeth touched them or hurt them.
I've seem too many lives ruined, on both sides- kids and adults. I've seen 6 month olds raped and beaten, I've seen 10 YOs pregnant by family members.
And I've seen grown men and women isolated and estranged, treated like scum, because a kid got mad at them, or thought it would be funny. I've also seen kids who are actually being abused blame someone other than the actual abuser because they're scared, or been threatened.