In the summer the Boardwalk IS a very nice place to go - they usually have vendors (art vendors is one I specifically remember) but they have different things going on. Not only on the boardwalk but walking up and down and going in and out of the "tinker" shops is kind of fun. They also have several mini-golf places the next block over (off the strip) that runs parallel to the strip.
The Boardwalk when it's not warm is boring, but it's fun in the summer. I'm not much of a beach person either but I love just going down and wandering.
There are, as Breezy_Carol said the museum in Norfolk and other things to do and see in Norfolk as well. There is a shopping place called Waterside and next to Waterside is the USS Wisconsin at the National Maritime Center that you can tour. My dad did that this past fall and said it was GREAT! And was a great opportunity (we have been on the ships many times) for people who wouldn't get to have access to the base now (post-9/11) to tour a real ship.
Here are some links for you:
http://www.chinfo.navy.mil/navpalib/ships/battleships/wisconsin/bb64-wi.html
http://www.nauticus.org/
Here's a website for things to do around Virginia Beach and if you go over to Outdoor & Sports and scroll down to the Ocean and Beaches there is a listing of different things to do around the beach area
http://www.vbfun.com/vbfun_2003/index.asp
I love this area a lot (if you can't tell) and when someone gives a negative, I give 4 positives if possible. Yeah - it's changed a lot since I was a kid and started visiting the area in the 70's but it also has A LOT of great things to do with your family.
Also there is the Forbes Salt Water Taffy factory and I believe they still give tours. You can find out at their store on 17th and Atlantic Ave. They have moved to a new factory from their location on Lynnhaven so I don't know if they still give tours.