DHM... yes, I noticed that they don't give you a lot of time on shore here. Really can't get to some of the "must see" attractions like Butchart Gardens, Craigdarroch Castle, or the Royal BC Museum (which takes several hours to do properly).
On top of that... we've been having the worst June weather. Cloudy, windy, cool, and rainy, but that's the Pacific Northwest for ya!
By all means come back and visit when you have more time to do it properly.
Fall is an absolutely spectacular time to visit here and we generally get some really nice weather at that time as well.
Lovely drive if you choose to do it that way. We have driven down to So Cal many times and just love it.