I'm a fan of Seaport Village too, although I haven't been there in a few years, so I can't say whether it's "gone downhill" was one of the posters mentioned. It's an open air shopping and dining area right on the water. It's not a huge place, but you could probably spend a couple of hours there. We like to go at night because it just looks really pretty (and we're usually doing something like the Zoo or Sea World during the day). They have horse carriage rides (or at least they used to) that leave from there too.
We like to shop at Horton Plaza. It's a pretty big mall, and we like to joke that whoever designed it must've been on drugs because the escalators are in weird places (not symmetrical like most malls) and it's almost like a multi-level mall. They have some of the typical department and chain stores, but they also have unique shops as well. The last time we were there we ate at Napa Valley Grille on the top level of the mall (I think this is actually a chain - I recall eating at a place of the same name in Providence, RI), and even if it is a chain, we thought it was excellent! We liked everything about it from the food, to the service, to the atmosphere.
There is a shopping mall in La Jolla that we went to years ago that seemed pretty good (we didn't have a lot of time), but unfortunately I can't remember the name!
Since you like history and culture, you may want to consider visiting the Cabrillo National Monument. It's right on the cliffside. Nearby, there is a very impressive cemetery, I think where they buried servicemen, that we stopped to look at.
Balboa Park sounds like something you'd be interested in, although I can't say I've visited any of the attractions there aside from the Zoo.