We've stayed at both several times each with our kids (more at Poly) and can add the following:
Poly: Transportation to both MK and Epcot is super easy most of the time. (If leaving after fireworks . . . that's a different story and sometimes we'd just prefer the bus!) The rooms are significantly larger at Poly. Room size and transportation options are really the main draws for us, in addition to the fact that our kids love the theme. Captain Cook's is the one quick service at Poly - we think it's quite good, but we prefer the atmosphere at WL quick service locations.
WL: Transportation - we've had some not-so-great bus experiences at WL but it's always hit or miss at lots of different resorts. I absolutely love the boat to/fro MK - so much more relaxing that bus or (in my opinion) even monorail. The rooms are smaller, but cozy and the lobby area is fantastic. Has a much more relaxed atmosphere for us because there is far less visitor traffic than on the monorail resorts (Poly always seems to have so much visitor traffic.) The new Geyser Point quick service is wonderful for quick service or appetizers, same with Territory Lounge in colder months and Roaring Fork is compact and in our experience, less chaotic than some of the other quick service locations. Love the photos of the new pool on the villas side and the pool slide at the main pool is great for tiny kids (our older boys loved it when they were 4 and 5 years old.) The splash pad at WL for little ones is also very tiny kid-friendly. The Poly splash pad is a bit busier with more older kids hopping in (again, in our experience.)
Also consider whether you want to be in the middle of the action (Poly) or have a retreat from the parks (WL.) Our kids tend to prefer middle of the action while my husband and I like the retreat! Right now, I would choose (and have chosen for January!) WL for that feeling of quiet retreat at the end of the day.
But with tiny kids, if transportation is a priority, Poly might work well for you. If you plan to take breaks during the park days, Poly will make that very easy.
Good luck - either is a great choice!