If you're interested in history, then you could fill your day without necessarily needing a car. If you don't mind hilly walks.

Although I've never done it, Signal Hill would likely be a 15-20 minute walk (significantly uphill) from where the ship will dock downtown. On the way to Signal Hill is the Geocentre. An interesting place. I'd also recommend "The Rooms" which might be slightly less of a walk. For the Rooms, you'd want to see what they're doing special when you get closer to the date.
By the Colony, do you mean the Colony of Avalon in Ferryland? It's probably one the better drives you'd have time to do. The lighthouse picnic there is quite nice.