I've done way early and way too late, and I like early better. I have a "packing corner" where I start accumulating stuff for the trip. I prefer to start about two to three weeks ahead.
We stay in condos, so take along some food items. I start putting the non-perishables there as I think of them, so I don't have a long list at the end. Also the things like guidebooks, ponchos, meal vouchers, coupons, etc go there early.
The clothes usually get packed three - four days ahead, maybe earlier if we have a busy week going. The last morning is reserved for filling up the cooler and packing the toothbrushes. I don't like to be rushing around right before we leave, though I still always have a list of last minute stuff. I try to assign these things to other family members the day before, but someone usually lets me down.
Sheila