My son has always enjoyed Universal Studios and Islands of Adventure. We were there when he was 18mo, 3yo, 4yo (a few weeks ago) and we'l be going again soon. I have older kids, also, and if we had of waited to go until Jake was 8 my oldest two kids would be out of the house.
At 4yo he can do almost all the rides, except for Ripsaw Falls, Dr Doom, and the big coasters. He's really at a great ago now for most everything. Last year he was still pretty cautious, but that all depends on the particular child.
If this is your one and only trip to Universal I would wait several years, but if you're planning on going on occasion I would say to go for it and do what the kids are able to do. You can do the baby swap for the things the younger ones are too little for.
T&B