Never stayed at the Poly, but it is very convenient to the Magic Kingdom. A lot of DIS'ers love it there.
Our family of 5 loved it at AKL, nice pool, good bus service even though the resort is far away from MK. Your family would be comfortable at either one. I would go to the "Disney Resorts" forum and see what is posted there. There might even be a "frequently asked questions" thread.
Also, go to the official web site for Disney and look at the pictures for each resort. I think both are great, so it might be a matter of personal preference or even what one has the best reates. Good luck.