My husband and I went many times without kids. We saw the mess of strollers in Magic Kingdom and agreed that we would wait until our kids were out of strollers to bring them once we had kids. Then the in-laws wanted everyone to meet at Disney for their 50th anniversary. And so we did. With a barely 3 year old and a 2 month old. The baby wasn't the issue - it was the 3 year old. We learned a lot and went back 6 months later determined to use what we had learned. Since then, we've been many times with kids.
At 2 months old, the baby was in the sling the entire time. The biggest issue was me being able to drink enough in the heat. I need to drink excessive amounts the first few months of breastfeeding and I was still during that time and it was May/June - so a bit warm. At 2 months old, baby was still on the excessive poop schedule. If I had waited until 3 months, that would have been so much easier. But all-in-all, it was pretty easy with a baby that young.
We went back when that baby was 8 months old. She was in the sling a lot but would go in the stroller for naps. She actually napped some - which is far more than I can say for the next child when he went at 11 months old. Whether 2, 8, or 11 months, I didn't have to worry about a schedule other than getting them to bed at a reasonable hour. They were quite portable and stayed in the sling most of the time. (The 11 month old was behind a bit and didn't care about walking for quite a while longer. 11 months would have been super tough with my eldest who was running well before then). They all breastfed, so that was super easy to carry around with me.
As for rides, there were many rides the baby of the moment could do. There were some that the baby couldn't, but child swap allowed the rest of the family to do it - me included. Baby didn't care about doing rides though so it was about me getting on rides too - which child swap made possible.
From going with both babies and toddlers, toddlers are harder. Toddlers can run. Toddlers might think it's hilarious to run away from you. Toddlers care more about what rides they do and don't do. Toddlers get scared more easily than babies IME. Babies are far more portable.
So, if I had to pick an ideal age for babies, I'd say around 4-6 months. They're pooping less frequently, nursing is figured out, you've recovered fully from the birth, and the majority aren't very mobile yet.