Which ferry route is faster depends on your timing. We wanted to get to St. John as quickly as possible, and because we didn't leave the ship until 8:30 or so, the taxi drivers at the port told us that we would get to St. John faster if we took the short taxi ride to the Charlotte Amalie ferry landing, and then the 45 minute ferry from CA to St. John, then the longer taxi ride to Red Hook and the shorter ferry ride. This was based on the departure times for the ferries: because we left the ship a little late, we could make the 9:00 at CA, but not the 9:00 at Red Hook (our first choice).
Coming back from St. John, we took the ferry to Red Hook, because the departure time was more convenient. Overall, I believe it's faster to take the Red Hook ferry and then drive across St. Thomas back to the port.
The taxi ride from the port to the CA ferry was quick and easy. We just approached the taxi line and found a taxi that was leaving for downtown. It might be worth renting a car on St. Thomas if you're going to be exploring more of the island, but our goal was to spend as much time on St. John as possible - so we went with a taxi.
I hope this wasn't too confusing! Keep in mind that the ferry to/from Charlotte Amalie doesn't run as often, or as late. You can find a ferry schedule
here. Have a great time on St. John!
April