My biggest piece of advice is to wear matching shirts! It is much easier to keep track of your kids and, if one wanders off, it is helpful in finding them. As an added bonus, it looks great in pics too! Obviously, you wouldn't have to with the baby. For our trip next month, I bought everyone red hoodies and am going to Disney-fy them.
And, no, you're not crazy! Our craziest trip was in 2006. We had gone from a family of 6 to a family of 10 in July with the addition of a sibling group of 4. Then, we went to WDW in September. The kids were 4, 4, 5, 6, 6, 8, 8, and 10. We were still getting to know each other and were learning how to function as a family of 10, and our kids have some special needs and were working through some behavioral issues as well. Even with all of that, we had a fantastic trip and I know you will too! No worries!
And - traveling with a baby is super easy! Jude was 7 months old on our last trip and I loved it! He just went with the flow and child swapped as needed. (I am much more concerned about our next one at 22 months.)