The argument that really strains my connection to the Buddha comes up, usually as some form of, "Times were different then and people thought they were doing the right thing." Something of that nature was actually in this thread's OP (I could go look ... it's right up there ... nah). As if 160-240 years ago slavery was just something everyone thought was fine. Which is bunk. Abolition was a popular political movement from the beginning of US history. Popular now in the flawed but wonderful musical about a flawed but ... well really just very flawed abolitionist A. Ham. It took Hamilton a while to come up to speed on abolition but he did, and so did a lot of people. England embraced abolition and where it could not abolish it outright, the crown purchased living slaves to free them.
The idea that this was just something that happened is a myth. It was always a divisive issue, useful for determining which side of the room had a soul.