We've gone the week before Christmas the past two years, and we're going again this year (counting down, 20 days!). We always have a great time, and get home in plenty of time for Christmas Eve with the rest of the family (DH hates WDW, so DS and I go for a mom-and-son trip).
Yes, you will miss the worst of the Christmas crowds -- the difference between Dec 23 and 25 is staggering. And since you won't be there during a weekend, all the better! All the same, it's pretty crowded. I'd suggest planning carefully, especially with regard to Extra Magic Hours: Either take full advantage of them (if you're staying on-site), or avoid the EMH park like the plague. All the other usual planning suggestions apply: Use Fast Pass, arrive early and take an afternoon break, etc. The standard advice in the Unofficial Guide to WDW and the TourGuideMike web site all apply, fer sure!