You will be facing out bound traffic in San Diego but it should be about 2 hours depending on traffic, idiots, the border check thing up by San Onofree and more idiots. Part of the way is car pool lanes and most of the way is wide open freeways unlike the narrow ones up in my area. Check street view for Google maps to see the lanes you need to be in for the trip. It is pretty easy driving once you are out of San Diego.
At midnight or noon you could do it in about 90 minutes or less if you have a heavy foot.