DH and I usually stay at the Monterrey Inn which is directly across the street from the Matanzas River/Bay in the Historic District. We've stayed here probably 20-30 times. It is a very basic, circa 1950s motel with older, basic furniture, but is clean and very well kept and is walking distance to everything in the historic area (you can take a tram or drive to the Alligator Farm, Beach, and lighthouse).
It's $99/night on weekends and less on weekdays. It's definitely the least expensive bayfront motel--the Hilton, almost next door, is at least twice the price. Most of the rooms have refrigerators and many have views of the Bay (if either are important to you, be sure to request them).
The Days Inn Historic is fairly close to the historic district but probably not what I consider walking distance. However, it's a good, clean, basic hotel in a safe area.