I am going to be different and say the Swan is a great choice! It is a luxury resort in every way. Sure you have a boat to Downtown Disney at POFQ, but you have a boat to Epcot and DHS at the Swan. It is so nice to go to theme parks where you don't have to take a bus with small children. At 4 she would still be comfortable with a stroller so that she won't get so tired. It is a bit of a drag loading a child plus a stroller on a bus, so it is a relief to have the boats or walkway available.
The Swan pool really isn't that far away and is much nicer than the POFQ pool. Our Granddaughters (age 2 and 5) love staying at the Dolphin! Our son takes his family to WDW every year and they almost always stay at the Swan/Dolphin or a split stay between an MK resort and the Swan/Dolphin. The pool area is awesome!
The rooms at the Swan are larger and more comfortable than at WDW moderates. Interior corridors are very nice. Also you can book a room with a balcony so that at nap time, you and your SO can go outside on the balcony with a book and favourite beverage to relax.
Then there is the dining.. You have so many more dining options at the Swan and in the Epcot area. At POFQ you don't even have one TS restaurant! Only the food court!
there are lots of great discounts available to lessen the difference in price, and you can often get a kids eat free with adults promotion.
In our experience, toddlers are happiest when the parents are happy. She will have a great time just spending time with you. Also, there is lots of stimulation in the parks so it is nice to go back to a quiet resort for rest and 'down time'.
POFQ is nice, but it doesn't compare to the SWAN, IMO.