I grew up on Vancouver Island and so am fairly familiar with Victoria. There's plenty to see and do there, it is a really pretty and quaint city. However depending on how big of a priority it is to go there for you I don't really recommend it as a day trip, unless you're willing and able to go early in the morning and return late at night, or are able to stay overnight. (if so I recommend trying local bed and breakfasts, it's a great way to go--and many of the houses are gorgeous heritage houses)
Let me explain:
My parents live on Vancouver Island, about half an hour from the ferry and I generally consider it takes me 4 hours to get from my place in Burnaby (which is half an hour(ish) from downtown Vancouver to put it in perspective) to get to my parents place. This is not total travel time, but you do need to arrive somewhat early to the ferry, even if you purchase a reservation (addtl $17-$20 above already exorbitant ferry fare depending on how far in advance you buy it) Also this reservation is only valid if you arrive at the ferry between 30-60 minutes prior to sailing. crossing time is 1 hr 35 minutes from Tsawwassen (30 minutes south of Vancouver or 1+ hour by bus) to Swartz Bay (30 minutes drive north of Victoria or 1+hour by bus), and about the same from Horseshoe Bay (30 minutes north of Vancouver, or 1+ hour by bus) to Departure Bay, Nanaimo (1.5 hours car drive north of Victoria--no public transportation other than greyhound) (all of my driving times are approximates!!)
Please note that there is no ferry that goes directly from downtown Vancouver to downtown Victoria. There is a ferry that goes from Port Angeles, WA to downtown Victoria, that I suppose one could take if you were flying into Seattle. I am not familiar with the logistics of getting there however, I believe it is quite a distance from Seattle. I wonder if there might even be one that goes direct from Seattle?? I don't know though.
Having said all of that, if you are prepared for the cost and travel time involved, public transit would be the cheapest way to go, and the buses are generally set up to coordinate with the ferries so getting to downtown Van and Vic is not too difficult in either case. Taking a car across is simpler, but cost of vehicle on the ferry is $47.25, each way.
There's lots to do in both cities. Victoria is set up to be more, well "Victorian" ie: has stronger British feel to it. There are double decker buses all over the place, in fact, many of the regular city buses are double decker buses, whereas, to my knowledge, the only double decker buses in Vancouver are tour buses. in the downtown Victoria area there are lot of horsedrawn carriage tours. There are also horsedrawn carriage tours in Stanley Park, Vancouver. Victoria has the parliament buildings, and a very very good museum if that is your thing. Within walking distance of these are whale watching tours, the wax museum (haven't been since I was a kid, but I thought it was cool at the time!) the famous Empress Hotel, which is cool to just walk through, and beautiful to photograph, and maybe even go for high tea (expensive). Walking along the harborfront is beautiful and there are almost always local artists selling their wares, (a comparable area near Vancouver would be Lonsdale Quay, which also has a market (north vancouver), accessible from near the cruise ship terminal via seabus, which I think is cool in itself.

another similar, but even larger area that I really recommend is granville island, which is accessible by public transit (personally have always driven there--if you go, look out for the glass blower and do a beer tasting and tour at Granville Island Brewery--awesome beers). Also in the Downtown victoria area across from the Parliament Buildings is the Undersea Gardens (basically showcases local sealife) but I don't recommend it, it's small and can be done in 30 minutes or less.
So the victoria stuff above is all in the same few blocks. Not far away, (but still within easy walking distance, is the very touristy area of Government street, which has a lot of quaint shops selling things like maple syrup, local knitted sweaters, tacky moose t-shirts, etc lol. But it is a fun and pleasant walk, and I recommend a visit to Mrs. Murchies for a great selection of teas and pastries. I know there's one over here in Vancouver area, but I'm not sure where. (I would say a comparable area to visit in Vancouver would be gastown, with it's historic steamclock and associated touristy shops. It is located a short walk from the cruise ship terminal downtown)
a little further away from the harbourfront in Victoria is Craigdarroch Castle, which is small, but kind of neat to go through if that's your thing..then there's Butchart Gardens (which is much nearer to the ferry than downtown victoria--but still a drive) Butchart Gardens are really fabulous, (but expensive) and if you can go during fireworks days (Saturday evenings in the summer, until mid september) i highly recommend it, But in that case you need to get there midmorning and stay until night time when it closes. The fireworks are choreographed and included in the entry. Also viewing the gardens by night with all the different colored lights is not to be missed if you go there. The cost of entry becomes totally worth it if you can go on a fireworks day and do all of the above!
If you are interested in aquariums, I highly recommend the Vancouver Aquarium, which is fantastic. (not as extensive as SanDiego Aquarium, our larger animals include seals, sea lion(?) sea otters, belugas, dolphins, small sharks, a local giant octopus and some very cool amazonian monster fish. I would say it's well worth the visit, and you should give it 2/3 hours to tour, see some demonstrations etc. The Vancouver Aquarium is located in Stanley Park, Vancouver, which in itself is a fabulous destination. I recommend renting bikes and riding around the whole park, which takes 1.5-2 hours if you go at a leisurely rate and stop to take pictures along the way. (it's about 12 kms (7.5 miles) around) You can even rent tandem bikes, which are really fun!
For those of you interested in turn of the century period type stuff I highly recommend the Burnaby Village Museum, we are local and love to go there over and over. It is an outdoor reconstruction of a turn of the century local village, in most cases the buildings are actual heritage buildings that have been restored and moved to the location. there are interpreters dressed in period clothing that give demonstrations of what life was like at that time, to be found in the printery, the farm house, the blacksmithy, among others. There is a little charlie chaplin theatre (also one at the victoria museum incidentally) a historical working carousel, and a cafe. Photo ops are abundant. Especially as this is attached to Deer Lake park in Burnaby which is beautiful.
I could go on and on but I think this is a long enough post!

lol I will add some links for info: hope they all work!
hope i made sense and sorry if this was information overload!!
http://www.bcferries.com/
http://www.royalbcmuseum.bc.ca/MainSite/default.aspx
http://www.translink.ca/ (vancouver public transportation)
http://bctransit.com/regions/vic/ (victoria public transportation)
http://vancouver.ca/parks/parks/stanley/
http://www.fairmont.com/empress
http://www.vanaqua.org/
http://www.butchartgardens.com/index.php?option=com_frontpage&Itemid=1
http://www.humboldthouse.com/
http://www.dashwoodmanor.com/
http://www.beaconsfieldinn.com/
http://www.ambrosiavictoria.com/
http://www.lonsdalequay.com/
http://www.granvilleisland.com/discover-island
http://www.burnabyvillagemuseum.ca/