We have stayed at both. Our first stay was at the Poly then after that was the GF and since then we always stay at the GF. The Poly was fine, but the GF is more our style. We love everything about the GF, the main building, piano player, the band, the pools, the shops, Narcoossees, Gasparilla, Mizners, what's not to like.
The Poly to us was too old and dark. Some people like that but not us. One nice thing about staying at the Poly was being able to walk over to the TTC and catch the Monorail over to Epcot.
I think you will be happier by sticking with the GF.
Just my opinion.
Tom...