I've actually done all with toddlers, so I might be able to help...
Swan - I chose this over Dolphin, beds are Queen Dolphin's are fulls... clean room, clean hotel, but not really "Disney", walking distance to Boardwalk that was fun!
LONG walk to Epcot. Restaurants in Swan are $$$ to get to the "cheaper" restaurant you have to go through the resort over to the Dolphin...
W/toddler - bus ride to MK is LONG to say the least, we found we were better to take the boat to EP (w/park hopper pass) go into EP, walk out to transport center and monorail to MK (yes, the bus ride is that long that it made all that worth it, often by the time the bus got to us, it was already full)... (P.S. can't depend on that way home for extra magic hour days)...
Pools at Swan - very unhappy... didn't use them... didn't go to Dolphin pool so no comments on that one...
I stayed at a downtown Disney hotel right across street from Hilton... buses are again LONG... long waits for them to arrive, very full! We once took a bus to downtown Disney (from MK) and then walked 2miles to hotel, bus lines were so bad! - However it fit in budget... again, not "Disney" themed...
If going back to hotel during the day is not a big deal, then save the funds for something else and go with either resort... if it is...then I suggest staying closer to or on property...
DD age 3 (both times) would have had more fun in a "Disney" themed hotel, she napped a LOT in stroller while we wandered parks, (I took a change of clothes each day to refresh her, made for LONG days!), I would have liked to be closer to go back and forth between hotel and parks... but it all worked in budget...
AoA has their own dedicated bus system, as does another Value, I'd look into those...
*I stayed at Poly in 2004, convenience out-weighed price, we traveled with a 6 yr old (nephew)at the time, went back to the hotel frequently, (having that monorail paid for the resort)... mid-afternoon because of heat, pool is awesome, then back to the parks for night time fun... Same with Beach Club, bus to MK but wasn't as bad... boat to Epcot or walking distance... spent a LOT of time on the Boardwalk...