What a disaster it was on Saturday when the Magic didn't get cleared until ~8:30am. Let me tell you, it was a mass exodus and long, long lines to get off the ship & through customs.
I don't think this happens that often, though.
When we took WDW transport from the port to CBR back in 2005, we got back before 11am.