Well, I have to say that both hotels are very nice. I've stayed at the Poly many times before so I knew that i would love being there. The Swan was new for me. The heavenly beds were wonderful (and we were upgraded to a balcony room free of charge). The view was very nice and the staff was extremely friendly. However, since I've been going to disney since I was a little girl, the monorail hotels hold a special place in my heart and for that reason we would probably not stay at the swan again. If you are a preson that spends a lot of time in MGM and Epcot, then the swan is perfect. We spend most of our time at MK so the monorail hotels are more convenient.
Couple of other points:
We asked to have the resort fee ($10) at the swan waived -- they do so immediately and did not even question us. The bellman there were much more efficient than at the Poly.
We ate at Palio -- oh my god - their food is incredible - I highly recommend it.
We had room 1315 at the Poly in the Fiji building -- incredible view -- I will always request that room when we go again. It was close to the Ceremonial house too.
Saw a preview of Wishes (they would show it just about every night after the MK was closed for a couple of hours). We watched from the Poly (and the music was piped in). It was truly a magical show. I was in tears watching and listening to the music.
As for crowds (we arrived on the 26th and left on the 8th) -- we walked onto just about everything the first part of the trip (except for the weekends at the MK) - Saturday there is just a zoo! As the week went by, we could tell crowds were starting to build. In fact, Sunday the 5th was very busy and we could tell the Poly was getting more full.
I'll post more if I can think of more.