I would want to gradually and gently transition each child into the correct home, over a period of a few months.
And then I'd ideally like to maintain a friendly relationship with the other family, so that each child ends up with something like a family and a stepfamily.
Before I had children of my own, I would have thought "family" was all about who's raised you, and I'd have voted for the babies staying where they are. But now that I've raised two teenagers, I can tell you that blood counts for a LOT.
Not to diminish adoption in any way, but I understand now why my best friend in highschool was so excited to get her medical records from her birth parents. She didn't feel any need or desire to "connect" with them, but it meant a lot to her - as a technology-minded girl growing up in a family full of artists and academics - to know that her father was a mechanic. She said suddenly she felt like her desires and talents made sense. She wasn't weird, she was just her father's daughter.
My children are a fascinating mix, not only of me and their dad, but also of their grandparents on both sides. It's really lovely to see echoes of their deceased grandfather in them, particularly my son, who seems to have got the larger dose of his father's side of the family.
So yeah... transition the children back. And hope like heck that the other family are decent folk.