Hiya,
I'm a local... come on out!

The PP who talked about the traffic is right, but there are tricks to getting around even on the worst of days.
I wanted to suggest Chick N' Ruth's Deli. aka-Chickenruth's. It is *NOT* classy, romantic, or any of those things... but it *IS* the most "authentic" little dive left on Main Street. We see a lot of places come and go... but this place is a classic. It's also kind of loud and crowded and brightly painted in garrish colors... but that's what I like about it. The walls are covered with pictures of all the famous people who have been there over the years, and menu items are named for the people they like-- or don't. My life goal is to get a menu item named for me there someday... just preferably not the baloney or chicken sandwiches... those are not compliments...
I'd second what the others have said about going to a B&B, there are several good ones. There's only one or two big hotels in the historic district (which is really what you want, from the sound of it) and they are chain-places... no real character. Consider a harbor cruise-- they do a good job of pointing out the little bits of local color you might otherwise miss and aren't expensive for an hour or two on the water.
OH! And I always like to tell anyone who likes the "quaint and charming" atmosphere to imagine the inner harbor and outer anchorages as they really were... Warehouses and Wh-rehouses

Remember the slaves being taken from the ships there-- not trying to ruin it for you, trying to give you a better idea of the "real" Annapolis... it's even cooler if you look under the whitewash! And if you stop in to Middleton's Tavern for a snack or a drink, go upstairs... that's where the "working girls" had their rooms... ;D
If you're physically able, there's a great volksmarch (ava.org) that starts at the Visitor's Center and points out all sorts of little tidbits... just an idea.
Have fun, and Happy Anniversary!