We've only been to HH and VB once each. Here are my opinions:
HH is a very nice, relaxing resort, and HHI itself offers a lot....great golf from what I understand (although we're not golfers), great restaurants, nice shopping, beautiful beaches, good chance to see nature (e.g., Dolphin watch), great marinas, etc. The only real drawback of the HH Disney Resort is that it's not right on the beach....they have a beach house, but you have to drive or take the shuttle to it. Still, a very nice resort in a great location.
VB is a beautiful resort, and the kids activities can't be beat. The beach is a little narrow but still nice. The Turtle Watch is a unique and interesting nature activity. The area around VB is mixed though.....some very upscale housing and some poorer neighborhoods. We didn't find too many great restaurants....I'm sure there are some, but it certainly didn't have the selection of HH Island. We were just there for our first visit and while we absolutely loved the resort itself primarily because it was very pretty and because of the kids programs, we were disappointed in the area. I don't think it would be my first choice to go to if it was an adults-only trip. (I might be spoiled....we used to spend a lot of vacation time at Sanibel Island where my Grandparents have a condo. It's beautiful, great restaurants, great shopping etc. I like VB better for the kids because of the kids activities--the place we always stayed at Sanibel had none. But, I expected the VB area to be more similar to a nice Florida vacation spot, and it definitely wasn't that.)
That's my opinion. Hopefully I haven't offended anybody. Like everything else, I think they each have their pros/cons, so it's all a matter of what you're interested in.