Here's an anecdote. Spring Break 2013 we were staying at BLT, renting points. We hopped over to Poly one day to swim. All was well and the Mojitos were tasty. My daughter and I walked over to the marina to get her a life preserver. The young lady behind the counter asked for my Key to the World card, typed a few things on her computer and ask "are you staying here?" I said no, we are at BLT. At first she said we were not allowed to swim there. I told her I thought it was ok since we are staying at a DVC resort. She walked into the back room, I think to talk with someone else, and a few minutes later popped out and said all was well. She gave us the life preserver.
My daughter is now old enough to swim without a life preserver. So since there is no gate at GF villas, from a practical standpoint I don't see anything blocking us from hopping over there. And from the official-looking DVC rules posted above, it appears I'm (amazingly for me) within Disney guidelines too.