I ate at the Seafood Hut last summer. not fancy at all, but my flounder was tasty! Get there early; the line will be around the building if you wait too long.
If memory serves, when you get into town, turn at the Christmas Shop and head towards the water. It will be on your left. It's a little white building, not on the water.