I haven't stayed at either hotel, but for logistics the Poly will be your best choice because of the ease of getting ot the Magic Kingdom.
We had a 5yo with us and he asked to go to the Magic Kingdom everyday and we took him at some point everyday. Also at that time of year they offered Little Ones Magic Hour. Its a few days a week where Fantasyland is open at 8am. My nephew got to ride the Carousel with Chip & Dale, Stich was on Dumbo when we rode. He played "knees toes" with Minnie and Pluto and played duck duck goose with Peter Pan, Wendy and Capt. Hook. We did this as well as ride anything we wanted in Fantasyland. It will be really easy to get to the park early from the Poly then return to hotel for an afternoon break or even have lunch back at the hotel rather than at the park. There are also reports of SAB being somewhat frieghtening for young ones, though perhaps a non issue for your little girl.
The comment about SAB closing early is a big flag for me. We took advantage of our resort pool being open until 11 or 12. It was actually soothing after a day in the park and a really nice way to end the day w/o the sun beating down on you.
Another thing to keep in mind is that the Poly is on the seven seas lagoon. Just about every night there is an electric water parade on the lagoon that you'd be able to enjoy from the Poly. We saw it while having dinner at Narcoossee's. Our 5yo loved it and so did I (I'm a bit older than 5). You can also take a boat from the Poly to MK, a nice option. We took it from MK to GF for dinner then from GF we hopped off to take a stroll through the Poly. We took the monorail back to MK for our resort bus (Disney trans rocks).
There is a kids club at the Poly, Neverland, that is apparently the largest and it seems to get rave reviews. Your older one just misses the cut off, but the it might be something to consider for the litte one. The reveiws I've read is many cry because they don't want to leave when the parents come to get them. I think that's about the best endorsement you could get from a kid!
The one consistent complaint I've read about the boardwalk area resorts is the lack of counter service (CS) restaurants. You'll have Captain Cook's right in the hotel open until 11pm. There is nothing at the Beach Club and you have to head over to the Boardwalk to get get to the Boardwalk Bakery.
We are planning a deluxe stay on our next trip. We are looking forward to a stay at the Animal Kingdom Lodge. Further away, but dedicated efficient buses should be just fine for us.
Finally, I've read about split stays. I'm not sure of the length of your stay, but generally on vacation I just want to be there, so hopping, unless I'm on a cruise and to ship is "hopping" or just one night (ie. I have yet to unpack) wouldn't be for me. I can't imagine that it would be ideal for a family of four, but perhaps that's my narrow view; lots of families report doing this.
Let's not hve a fight break out though! I'm in your corner if it does!