We went during the same week (Dec. 17-22) in 2005, and we went Dec. 19-23 in 2004. Really, it's the perfect time. But yes, it gets busier toward the end of the week.
Be sure to catch the tree lighting ceremony at Epcot. It's around sunset every evening (check the time guides). We just happened upon it, while walking around the world showcase, and I thought it was beautiful!