It would depend on why I was going; is the trip for the family, adults, or the child? Also, the age of the child, their attention span, how easily they scare, and their like or dislike of the Universal characters and ride themes.
There are plenty of attractions to keep young children occupied, but not enough that I feel they are worth the cost of the ticket. There are a few rides and shows if the child has the attention span to sit through them and doesn't get scared, but most of the toddler attractions are playground / play areas / splash areas. Most of my favorite parts of Universal wouldn't have the under 4 crowd as their target audience, and I wouldn't want to drag a small child through one rider swap experience after another. Not if the trip was for them.
So, if my whole family really wanted to go to Universal, I wouldn't not go because one of the children was young. We could find things to entertain that one child. For instance, family of 5 is going, two older kids, mom, dad, one toddler. Everyone is really excited about going and seeing everything. I would definitely go and use rider swap and make the most of what is offered for the toddler. I think everyone, toddler included, would have a great time. In this case, the trip is more for the rest of the family, not the toddler who is really too young to understand what is going on anyway.
However, if the trip was more for the young child/ children specifically. For instance, mom, dad, 3 kids all under 4 and the trip is about letting the kids experience as many rides as possible and spending time together as a family, not splitting up, I'd pick somewhere else. It wouldn't be worth the cost of admission to me as we could go somewhere else geared more toward that child's interests and abilities. My personal opinion is Disney,
Legoland, Busch Gardens with the Sesame Street section and animals, or Sea World with its animal interaction would be better options for most families with very small children. Exceptions would be if the kids just really wanted to do or see a character or something at Universal that they are really into.
Best advice I can give you is to make a list of the attractions your kids will be able to do, and then figure out how many of those attractions they would actually be interested in doing and then decide if that is worth the cost of admission. Then ask yourself does the rest of the family want to experience all the other rides the kids can't or won't want to do, and if so, will you be ok with splitting up or making the kids wait in lines for rides they won't be able to actually ride.