Sure, I think the SD would make a nice day trip. It should take around 1 1/2 hours each way driving if you don't hit too much traffic (allow for 2 hours). I'd suggest getting a fairly early start so that you can do the tram tour early. I'd guess about 5 hours would give you a chance to do the tram and leisurely stroll most of the park, although not all the more remote parts. Of course, it wouldn't hurt to allow a little more time to visit the gift shop. You also might find yourself tempted to wander around Balboa Park a bit too.
It would make for a nice, fairy long day, but you'd get back to Anaheim in plenty of time to get a good nights rest.