Border lines are a bit like visiting the parks. Everyone knows that you need to get there early, but most people don't seem to do it. I live a few hundred yards from one of the border crossings and even on long weekends the lines don't really start before about 7:30 or 8:00. If you're willing to hit the road super early (like 5 am early) you'll probably be fine. Labour Day is probably one of the best mornings to drive in the city. Everyone gets the day off, but there's nothing really happening. The tourist-y things will be busy and the area right around the cruise terminal may be crowded, but getting in and out should be okay in the morning. Watch out after about noon, though. The routes back to the city will be a parking lot in the afternoon and evening as everyone goes back into the city from the lakes and mountains. People here take their long weekends seriously!