Just putting my 2 cents in. I have both a custody agreement and passports, and the 2 times I have taken my kids across the border, I have obtained a very detailed notarized letter, signed by their father. My kids are 7 and almost 3, one time I travelled with both, most recently just myself and the older one.
I'm sure it varies where you cross, and which border agent you happen to get, but both times all I've done is shown the passports, answered the question about where we are going, and was waved on. They never even asked about custody or travel letter!
Better to be prepared, but don't be surprised if they don't ask!