VB resort (the areas of the building you can access without staying there) is pretty small. My in-laws had a house in VB. We stayed there a few days and ate a breakfast at the Disney resort.
IMHO, VB is a pretty boring town (at least it used to be when we went about ten years ago). It's ok if you like to sit on the beach all day, but there's not much else to do. A weekend there would be enough for me.
If you are looking for a day trip from Disney, you might like it. It's about a two hour drive, so budget four hours for driving.