We have gone to the port from WDW many times in the morning and have not had traffic. I do avoid I-4 at all costs, though. I take 417 to the Beachline.
I like to be at the port when it opens at 11:00 and you will have to check in at the port before you board so also aim to be at port by 11:00. Leave WDW about 9:30-9:45 to give you a bit of extra time.
MJ