Definitely do Poly at that price difference! You'll be so glad to be on the monorail and skipping buses for MK & Epcot. (Now, if you're headed to AK, HS or DTD you'll be riding buses, but there's not much to be done about that!)
Poly is such a fun resort--we loved exploring the beach, the GCH, watching the MK fireworks across the lagoon and the parade on the water at night. The volcano pool is really fun, and I loved the food there. Captain Cook's has Dole Whip all the time!

Since you're staying there you may as well eat breakfast or dinner at Ohana, too!

Everyone seems to love Kona Cafe, too...but I didn't make it when we stayed there. It sounds like AofA is a really fun resort, but both you and your kids will love Poly!