Traffic can significantly increase your drive time. If you are going on the weekend, you may still hit traffic but it will be less of an issue. You can drive there and back whenever you'd like.
On weekdays, you will definitely be affected by rush hour traffic -- especially if you are trying to get to Universal by opening time. I would allow an extra 45 minutes (on average) in the morning. If you don't mind going a bit later, traffic can get better if you depart after 9:30am from Anaheim.
Coming back from USH to Anaheim, I would depart no earlier than 7:30pm. Even during the off-season, Citywalk is open late so it's not a bad idea to catch dinner and wander the complex until traffic gets better.
It's definitely worth making the trip to see Universal. Have a great time!