We were there last year for 1 week beginning with the Saturday before Labor Day. The crowds were not that bad at all. We did EE at the MK that Sunday and completed all of Fantasyland within the first 1 1/2 hour. The crowds did pick up as the day went on, but it was not as bad as the usual summer crowds. I don't think we waited more than 15 mins. for anything.
We went to AK on Labor Day and barely had to wait for anything. They weren't even giving out fastpasses for Kali because the standby line was only about 5-10 mins. The rest of the week was pretty empty. A CM told me that is one of their slowest weeks of the year.
We liked this time of the year so much, we're going back again this Labor Day weekend!