Yes, you can use the GF's fitness room which is in a free standing building with the Spa. It is almost as close to some of the Polynesian buildings as it is to the GF.
It is free to guests of the Polynesian and GF these days. As you said, just take your room card.