People have often asked this. When you go on the Members site and click on the Pool Hopping, this is what it says:
Whenever you are staying on your Membership at Bay Lake Tower at Disney's Contemporary Resort, Disney's Animal Kingdom Villas, Disney's Beach Club Villas, Disney's BoardWalk Villas, Disney's Old Key West Resort, Disney's Saratoga Springs Resort & Spa, Disney's Vero Beach Resort, or The Villas at Disney's Wilderness Lodge, you and any Guests staying with you, are permitted to use most of the other Disney Vacation Club and Walt Disney World® Resort hotel pools (in addition to the pool at the resort hotel at which you are staying) if they are not at capacity
Now this mentions VB, and it seems to me that people have posted doing exactly what the OP is asking. However, during the summer months VB's pool is probably at capacity.
The beach is free to anyone, and there's a small state park beach adjacent to the VB property that has lifeguards, so you may feel safer swimming just north of VB.
Bobbi