I don't think you can go wrong. We chose the Poly over GF because the theming seemed better for our family. Personally I would love to stay at GF, but not with kids. It's not that it isn't kid friendly (we visited several times with our DDs), it's just that there are more adult things to do that I would want to be doing and would spend part of my trip wistful (would love to do some of the dining with just DH, linger over high tea (though we did take our DDs there and it was great, just not as relaxing for us), have cocktails in the evening, etc., etc.). At the Poly I never felt like I was "missing out" on adult time. Everything we did was totally kid friendly and not something that I was wishing I could be doing as an adults only activity. Also, if you plan on going to Epcot, Poly is great because you can walk to the TTC and grab the monorail to Epcot. Transportation is equally as easy to MK. You can take the express monorail from TTC, the water launch to TTC, the resort monorail (one stop at GF before MK), the resort water launch (no stops), so you have a ton of options. I wouldn't make your decision based upon ease of getting to MK, because it is equally easy from either the GF or Poly.
Really, it's just which themig do you like better. GF sort of transports you to another time and place -- more refined, I think (and I'm *not* using that as euphimism for stuffy). Poly is more laid back, very tropical. Both are great.