Probably two or three, minimum, for your first trip. It depends on your goals - do you want to do nothing, or do you want to do some exploring?
Our shortest stay has been 1 night and we had a lovely time, but we were VB veterans at that point.
We find VB after WDW works best - it gives us some time for R&R before returning home.