We have never been to the Ardastra Gardens (actually, I've never even heard of them
)and not even sure where they are, but I can tell you that you can comfortably tour the pirate museum in an hour. We did tour the museum and didn't rush thru and we were done in a little under an hour. We walked to the museum from the port and it was less than 15 min (prob more like 10 min). I would highly recommend going across the street from the pirate museum and checking out Christ Church. The stained glass windows and pipe organ were phenominal to see. It's very close to the pirate museum and there is no tour of the church. You just go in and check it out yourself. I thought it was pretty incredible.
Good Luck w/ your plans.