We drove to San Diego multiple times in our last trip and it was fine. I would leave early no matter when you're going, or try to plan to avoid rush hour traffic. We didn't stay in SD, we went back to Anaheim each time. There is construction on parts of the I-5 going towards LA, but we managed to avoid it and never had any problems.
I think it really depends on the age of your kids. We have an adult son, and we all really enjoyed Seaworld. We got there late on a Saturday because we left late and there was some traffic going there. (Ended up taking a somewhat scenic route to avoid freeway traffic.) We saw all the shows and exhibits (except the Sesame Street one) and didn't bat an eye about missing the rides. Would I have liked to ride them, yeah, it looked fun! But the shows were incredible. Big respect for the trainers, whales and dolphins. It took us all of an hour and 45 minutes to get from Anaheim to San Diego and vice versa. We went to Barona Casino one night too, so we did this trip twice.
If you have kids-young adults in your party, everyone will enjoy Knotts and it's practically right down the street from DL. I highly recommend Knotts especially for teens who like thrill rides, but they also have a camp snoopy for young kids.
So that's my take on it. You don't have to stay in SD if you are used to driving far distances and have no issue with driving, at most, a couple of hours back to Anaheim. If you are not used to long distance driving and the LA freeway system, or are not comfortable reading a map, or do not have a GPS, you may want to rethink this idea. We didn't have a problem with it.