Hi! Long time lurker but this question is near and dear enough to my heart that I thought I'd finally register.
I've been to WDW a zillion times and now only stay at the Swan/Dolphin (I'm in consulting so I travel more than I like to think about but the Starwood points are great to either bank or use) after staying at all the deluxes in the Epcot area and on the monorail. Here are my thoughts:
I think that of all the WDW pools I've experienced, the pool at the SD can only be topped by the YC/BC and that the pool at the Poly is overrun with poolhoppers. Until they enforce that, its really not fun being overrun with people that shouldn't be there.
I hate WDW toiletries - and did hear that S&D finally moved to Molton Brown, but even Bath and Bodyworks are better
The location is a bit better from my POV. You can walk or take the ships to Epcot or HS and if you want to take the little one to Wishes one night, you can even go the LONG way and go to Epcot, grab a drink for mom and dad and then take the monorail over to MK.
The resort view balcony rooms can often be had for a nominal onsite upgrade fee at the Dolphin and the view of Illuminations is much better than any view I've had at the Poly (but not as good as some views from CR - I'm sure Bay Lake is even better, I haven't done any of the DVC stuff, I like hotel rooms, not condos, my mom and SIL are the exact opposite, I love that WDW can give us all our ways
The food choices are the best for convenience and quality (especially if you include the entirety of the Epcot resort area). Yes, this means you have to either do the WDW transportation hokey pokey or take a cab to do V&A (which you would probably be skipping with the little one, right? I know we are on our next trip in Sept. with a 5 year old!) or Cal Grill, but the ease of the counter service coffee station in the mornings to the variance of offerings is just not to be missed. Honestly, we used to do Cal Grill every trip, but now with Kimonos (agree with previous poster on it being GREAT!) having better sushi and Cal Grill's movement to a bit less innovative menu, I'd take Flying Fish or even Kouzzina over it. Also, if you can't get a reservation (and I will totally admit to tipping CMs for a table!) somewhere, I've found the availability of Boardwalk restaurants and S&D restaurants a bit manageable. I don't have the luxury of planning things 180 days out to my job so I'm always scrambling to get in where we want to eat - although a lot of our faves must not be very popular because it has seemed easier lately.
The beds at the S&D are much more comfortable than anywhere else I've stayed at WDW.
The fact that it's a little less themey and a bit less loud are serious pros for me but I know lots of people want the towel animals, etc...to each their own!
We rarely go for more than 3 or 4 days at the most (I like to tag a short WDW stay to biz meeting and go more often for a short duration) and park hoppers are the best, especially in August when it still be a bit crowded. I can't tell you how many times we've decided to leave a too crowded park (even with the crowd calculators now) based on what the ride times are in another park - and BTW, god bless all those new and amazing apps - WDW Waits is my favorite. You guys let me know if you know of a better one.
Anyhow, either way you will have an amazing time and I've learned a lot from this board!