I grew up down there. It can be pricey. Forhotels, we always stay in the Super 8 in Groton--it's just a couple miles form the sub base and Nautilus museum. For ~$50 a night, you get a decent room and free breakfast. There are a umber of cheap fast food type places right there, and a grocery store and Super Wal-Mart (there's a fridge and microwave in the room). So, if nothing else, you could have cheap snacks and the free breakfast.
A locally favorite restaurant is the Seahorse in Noank. A bit out of the way, by a huge marina, but it has less of a touristy feel than downtown Mystic. But, to be fair, I'm not such a fan of downtown. Another thought--Mystic Pizza IS right in downtown Mystic, and is the place from the movie, so that could be fun and inexpensive. In warm weather, Abbott's for lobster--sit out on the dock and watch the boats go by. Mind the seagulls!
I also love poking around Mystic Village--again, not cheap, but looking is free. Strolling downtown, also free.
In addition to the Nautilus, they used to have the Croaker (sorry, it's been a few years). It's an old WW2 diesel sub. It's an interesting contrast to the Nautilus, which of course is a nuke.
Also fun, if you're driving, on the way in or out of the area--Gillette's Castle. Probably not worth a special trip, but if it's on your way.