In the comics, they raped his wife and daughter (and several other women who they were hiding in that grocery store with), and made the son watch because he tried to stop them. When Abraham found out, he and several others killed them, but Abraham was the most brutal about it. And the rapists were neighbors of theirs, he mentioned that one was a kid who used to mow his lawn. This was still early on in the apocalypse, and the family was still in shock over the attacks, and even more afraid after watching what it turned Abraham into when he killed them. So the wife took the kids and got them away from him. Also in the comics, Abraham found the wife and son killed, but the daughter had turned into a walker. He’d been forced to shoot her. He actually told that whole story to Rick after the incident with the Claimers, when they tried to attack Carl (there’s no Daryl in the comics, it was Abraham that Rick was with when that happened), saying he’d done something similar when his own family was attacked. He told Rick he’d lost his family over it.