Between my 3, I've taken kids at 16mo, 2.5, 3.5, 3.75, 4, 6, 7, 8, 9, 10, 11, and 13. There are pros and cons to every single age and stage.
I loved taking my youngest on her first trip when she was so little (16mo). Everything was absolutely real to her, but aside from some shyness with strangers (including characters) she wasn't afraid of anything and was absolutely in awe of the magic. It takes slowing down a bit to enjoy the world at toddler pace but we thought it was well worth it and didn't even end up splitting up as often as we'd planned because the older kids (8 & 11 at the time) didn't want to miss any of the fun with their little sister.
My middle child was 4 on her first trip and that was far more of a challenge. She was at that too big for a stroller/too little to handle all the walking well stage, she was still new enough to potty training that we had some moments of getting out of line because she decided she did have to go after all (and Right. Now. of course), and she was fearful when it came to dark rides. traveling with a toddler was much easier.
The oldest was 7 and that was a great age from a rides standpoint - he rode everything with no fear, and he'd already been to Cedar Point so he wasn't worried about anything Disney could throw at him - but it wasn't a great age for the rest. He didn't have much patience with his sister wanting to meet characters, wasn't interested in most of the shows, and had some very strong opinions about some Disney characters and experiences being "for babies".
But personality plays into it too; my older daughter at 7 was nothing like my son at 7 (he was "too cool" to meet Buzz Lightyear, she was begging for another BBB princess makeover), and my younger daughter at almost-4 is nothing like older DD was at 4 (younger is fearless and disappointed she's too small for Splash, which older didn't get brave enough to try until she was 9!)
As far as resort choice, I like the values with smaller children. There's only one bus stop, none of the rooms are a long hike from the main pool or food court, and they're lively/colorful with landscaping and decor that invite hands-on play. Alternatively, the MK or EP area deluxes are lovely for easier access to the parks and again have the more compact layout that is easier on little legs.