I think this really depends on what is important to you, and how you vacation. They are both great resorts, but I would choose the Contemporary.
1.) Location
You simply can't beat it. It's a short walk to MK (and you cant imagine how satisfying it is to bypass the long monorail/boat lines at park close and walk home). It has monorail access, allowing you to hotel hop on the loop and you're also able to monorail to EPCOT easily. It also shares a boat loop with Wilderness Lodge and Fort Wilderness should you feel like going out to Hoop Dee Doo Revue or something at the campgrounds.
2.) Rooms
The rooms in Contemporary are huge. I can't remember if the Poly beats it in terms of sq feet or not, but the Contemporary rooms are not only big, but well appointed. It feels modern and new while still being comfortable. Very nicely appointed. And every room has a balcony with a great view. Even if you get lake view instead of MK, it's still very pleasant.
3.) Food Convenience
I don't know what age groups you're traveling with. But most people dont realize the options the resort has. California Grill is my fav restaurant in WDW and a must do for me, my big splurge meal. Chef Mickeys is fun character brunch for the little ones. The Wave is a hidden gem, its nothing particularly special but you can always get in w/o a reservation and they have a great breakfast buffet. There are also two lounges (one connected to Wave, and The Outer Rim). I also love having a "grab and go" breakfast counter to get my morning coffee and muffin (something many resorts lack) at Contemporary Grounds. Plus with the aforementioned monorail, you can easily get to the many restaurants and bars at the other loop resorts: Ohana, Trader Sams, dole whip at Pineapple Lanai, Kona Cafe, Narcoosee's, Citricos, Mizners Lounge, Trader Sams etc. The convenience of transportation here means you get to greatly expand the "world" of your vacation.
Cons
1.) Pool
This is really only if you're traveling with small kids who want elaborate pool themeing. The pool here is by all means fine, but certainly doesn't have the "wow" factor of other resorts with enormous slides or volcanoes etc.
2.) "Kid Appeal"
I always travel with adults (kids at heart!). But I've been told by many friends who are parents that the resort lacks appeal for their kids, based on the theme. Of course, to me the theme is perfect. I love the striking look of the A Frame building, the gorgeous mural in the grand canyon lobby, the way they light everything at night. But certainly, many of the other resorts have themes that are more fun for kids.
However, you have to take all this info in and apply it to your party and what is important to you. I travel to Disney with adults. We are also from NYC, and as such I'm used to a certain pace and I have zero patience waiting around for Disney's bus times. It drives me insane. So the convenience factor of the Contemporary is huge for us. Dining is also a big part of the trip for us, so its always great to be able to make restaurant reservations at the other monorail resorts and not have to worry about getting to them on time.
I would also agree with the above posters that Wilderness Lodge could be a good middle ground. It's not on the monorail, but its a boat away from Magic Kingdom, Contemporary, and Fort Wilderness. You could also transfer from that boat ride to the monorail, or take a very short/cheap Lyft ride to TTC or monorail hotels. The lobby is nearly identical in layout/construction Animal Kingdom Lodge (just obviously with a different theme) so it has that similar grand feeling to it. And while it doesn't have safari animals outside, the theme at WL truly does transport you to a different world. You can sit in front of the lobby fireplace, or out by the lake at Geyser Point Grill, and feel like your're in the pacific northwest. Whenever I think about doing AKL but balk at the transportation downsides, I come back to Wilderness Lodge. I recently stayed in one of the Copper Creek studio rooms and it was honestly my favorite room experience at Disney thus far. I know that adds a whole third option to your query, but it sounds like it could be the middle ground you are looking for.
