I'm not sure which direction you're coming, but if you a going in the next few weeks, double check to make sure if the Hood Canal Bridge is open yet before heading to Port Townsend!
Lot's of stuff is within walking distance in Victoria. The downtown area is right off the ferry. The museum, the wax museum (I've never been there), the Empress Hotel, the Harbor area. The only tourist "biggie" that's not right there is Buchart gardens and you can catch buses to it. I've done Victoria as a one day trip from Seattle or Port Angeles several times. It is also easy to stay over without a vehicle because there are hotels you can easily walk to as well.
I'm curious what will happen to all the daytripper business with the new passport requirements.