If your going in the summer, get your duck boat tickets on line before you go...
I'd recommend staying somewhere outta boston, salem, plymouth etc.. or the north shore. Its not that hard to get into the city, and in reality, will be much more relaxing for you if you make it "day trip" into the city for the aquarium, swan boats etc. We've done it staying in boston (near Mass General due to moms doc appts) staying outta the city, and then just day trips from southern vt.
Is Ocean Spray still in Plymouth? I remember going there and doing the big company tour when I was little...and of course, the rock. Salem is a quaint place, might not be interesting enough for a 4 year old..not sure
I probably luck out in that I can drive around Boston with no issue's since I've been doing it since i was 16 but its really not that hard to navigate if you do not want to use public transportation.
One thing to remember, the ocean is pretty darn cold in New England, so dont expect to be swimming in it for 2-3 days. Your ankles will be frozen pretty quick just standing in the waves!
Brandy