DEFINITELY GO FOR IT!!! We did this December 2004 and it was one of our best trips (well, all of them are wonderful)! I worried about the same thing, but we had such a blasting hitting all of our favorite spots. We wanted to go one last time before I got pregnant, and we had a blast. We used Fastpass really well to see all of our favorites. We knew we'd be back for future trips, so we didn't mind not seeing everything.
We arrived late Friday night and stayed at the BWV. On Saturday, we walked to MGM, spent about 4 hours in there, then walked to Epcot and stayed there the rest of the day. In addition to hitting all of our favorite attractions, we squeezed in a nice sit down dinner at Chefs de France, as well as the Candlelight Processional. We went to DTD for an hour that night (had to go to World of Disney and the Christmas Shop!). We spent Sunday in the MK until we had to leave for our ride to the airport home--flight was 8pm, I think.
We packed what we needed in 2 backpacks (I'm usually an over-packer, but it really helped to pack light for this trip) so we didn't have to check bags. We brought the back packs into MK with us, stored them in a big locker for the day so we didn't have to go back to BWV to get them. When we left MK, we walked to the Contemprary (5 minute walk) and we got our Tiffany Town Car there (arranged with them ahead of time to have a different pick up location Sunday than our drop-off on Friday). While it wasn't a laid back vacation because of the timeframe, we had such a wonderful time. We plan to do this for our anniversary later this year because we don't want to be away from our daughter for more than a weekend. I would highly recommend it, especially if it's just you and our hubby!