I've done this drive almost every year for the past 13 years, and it's generally not that bad. We usually take about 4.5 hours to get to DL from Vegas, including a couple of bathroom stops along the way. However, we do plan carefully. We usually leave first thing in the morning, about 6 a.m., and get to DL by about 10:30. Less people on the road then. Also, we try to avoid weekends. Keep in mind that a lot of CA residents drive to Vegas for the weekend, so they'll leave Friday afternoon. I got caught in that traffic once, and it was bad, but I don't think it added more than an hour to our drive. The weekend visitors will usually return sometime on Sunday, so I would also avoid leaving Vegas on a Sunday if at all possible.
Usually we make fewer stops on the way to DL from Vegas, so we always take longer returning to Vegas only because we stop and have lunch, maybe shopping along the way, etc.