I work in Virginia Beach and very much second Whitners. I go there at least once a week for lunch (and everyone knows when someone's gone to Whitner's at the office because you smell like smoke/delicious BBQ afterwards). It's a counter service type of place with about a dozen tables.
A good breakfast place is Citrus, but it is almost ALWAYS super packed.
Rigoletto's is a good Italian bakery/café, but they also serve breakfast, lunch, and dinner.
All three of these places have been featured on Diners, Drive-Ins, and Dives.
Another place that is close to the oceanfront and is delicious and family-friendly is Beach Bully BBQ.
The Neptune Festival is pretty popular, but I don't think it's crazy packed. Since Virginia Beach is a tourist destination and there's just about something going on every weekend down there, it's always crowded. But not overwhelmingly so. If you're in walking distance to the boardwalk from your hotel, that'll save you a lot of parking issues. Most of the parking garages on Atlantic Ave. fill up fast and most of the independent lots charge $20-$25 to park (and squeeze every car in like sardines). There's better parking options along Pacific, but you may have to walk a little further.
What direction are you coming from? I live in Hampton, but drive to work a few times a week (I also work from home as I was doing that drive 5 days a week for years. Ugh.) Thursday afternoons are just as bad as Friday afternoons around here. Usually back-ups can start happening at the Hampton Roads Bridge Tunnel around 2 p.m.-3 p.m. and get worse as it gets closer to 4-6 p.m. Usually by 7-8 p.m. everything has cleared up (unless it's been one accident/disabled vehicle after another).
If you take the Monitor and Merrimac tunnel, it's usually the best of the two tunnels (shipyard traffic can start around 3-4 p.m., though), but then you have to either go through the Downtown Tunnel through Portsmouth/Norfolk or over the High Rise Bridge in Chesapeake to get to Virginia Beach. Both of which have their fair share of congestion every day too (usually 3:30-7 p.m. is the worst). Oh, there is a new South Jordan bridge that's a toll bridge that you could take. They mail you a $3 bill if you decide to take that, though.
Are you up for renting a helicopter?
I'm kidding! Some days are better than others, in regards to traffic. Here's hoping you can get to Virginia Beach as smoothly as possible!
Oh, and if you're coming down 64 East sometimes there's a few mile backup around Ft. Eustis Blvd. in the afternoons. There are back road type ways that are easy to get around if you do some advanced planning. Once you pass the Camp Perry exit on 64 East (a few miles before the Busch Gardens exit), I would suggest turning your radio to 1680 AM. It's the most up-to-date with traffic issues and maybe have a few alternate ways to travel in case there's major issues on the route you were planning to take.