I would cut back on the carry-ons, if possible. The baby doesn't really need his own backpack and it's just one more thing you'll have to keep track of. Bring as little as possible. Going through security with kids and lots of stuff is very stressful. I have three kids and even though I feel I'm organized, going through security is still tricky. You have to take the baby out of the stroller and fold the stroller. Both the stroller and the car seat have to go through the x-ray machine, along with the bags. Plus, remember that everyone has to take their shoes off when going through security. When you walk through the metal detector, you have to get all your stuff, set up the stroller, put on shoes, etc.
My older two kids have backpacks with their toys, books, snacks, etc. I carry the diaper bag as my carry-on. I put diapers, wipes, formula, toys, changes of clothes for baby and me, etc in there. My DH has a carry-on with anything else we need to bring on the plane (DD's meds, books, magazines, gum, etc) in it.
Don't stress and have a great trip!