OK, I'll state for the record that I don't typically buy into rumors.
But the one concern I would have about owning all of my points at a non-WDW resort is the possibility that VB and HHI could (emphasize COULD) be sold off at some point.
Disney Vacation Development owns the resorts. Disney Vacation Club is the timeshare managment company. It's always possible that VB and HHI could be sold-off to another management company if Disney were to decide to divest themselves of off-site properties.
Many will say the odds of this happening are remote, and personally I agree. There have been rumors in the past that Disney would part with VB and HHI...obviously unsubstantiated. But 40 years is a LONG time for business plans and management philosophies to change. Even if Disney were do decide to get out of the timeshare business altogether, by owning at one of the Orlando-based properties, you're insured of a room on Disney property. But, if Disney were to sell off (or at least farm out the management of) VB and HHI, you'd lose the benefit of staying at one of the WDW resorts.
If your intention is to use the points at a Disney World resort, it makes sense to own one of those properties.