If it is just as easy to get to one as the other from where you are staying, I would go over the GWB and head down. If you do it this way, you are exiting to the right to get into the port area - so basically only dealing with the traffic on the West Side Highway. If you come up from the Lincoln Tunnel, first you are going several blocks into the city and have to come back out those few blocks to get back to the West Side Highway (can be a bit of traffic there that slows you down). Once you are there and heading towards the port you have to make a left across the highway to enter the port area. The light that you cross at can get very backed up (the one time our driver came up from the tunnel we sat at that light for almost 1/2 hour as it seemed that only a couple of cars crossed each cycle and we were quite a few cars back in traffic!)
