We find plenty to do! We had a 2 day 2 park pass with an 11 month old, 3 year old, and 6 year old and spent 1.5 in IOA and 0.5 at US - that was right before Harry Potter opened up. There was a lot more at IOA than US for them. We went again, but just for one day since it was supposed to be a Disney-only trip, with kids 2.5, 5, and 8 - and we easily filled up the whole day. It helps that all three love Harry Potter though. My youngest was probably 36" then. My oldest doesn't like thrill rides, so that made it easier. (Though we did get him on FJ. Once was enough for him though). My 5 year old would go on anything she was tall enough for, but she was under 48". One great thing about the child swap at IOA/US is that the child can go through the line too - so they could see all the theming.
We have 7 day passes for our next trip for IOA/US. We're waiting until next winter since for that many days, we wanted our youngest at least 42" and our middle 48". For just one day at IOA, or even two days, there was plenty for us. But for longer, we wanted them a little older/taller.
**As for what my kids do at IOA when 42" and below: Harry Potter- they love the stores, love the Ollivander's Wand shop and the other shops there. There's also Flight of the Hippogriff. (And Three Broomsticks is a great place to eat). Spiderman is 40" so my older two did do that. There's Me Ship the Olive, which is a playground they enjoyed. (Two have sensory issues so the three main water rides are out). There's Jurassic Park - the Discovery Center, Camp Jurrasic, the dino feet that make noise are fun. (There's also Pteranodon Flyers - I can't remember the height min but it's not worth standing in line for). They enjoyed Poseidon's Fury and loved the Sinbad Stunt Show. Suess Landing has a playground, if I ran the zoo. There's a carosel, One Fish, Two Fish (though the water squirted too much for one of my kids, but that water is mild), The Cat in the Hat, the High in the Sky Trolley Ride. Also, there was character story reading that was awesome. They were up close and had zero lines and tons of character interaction at it. The Mystic Fountain is lots of fun too - it talks to you. I could spend an hour just there easily since it's so much fun! There was also a little shop that has a little magic show they enjoyed - though I can't remember for certain if this was at IOA or US.