I haven't thought about the downside to
DVC because for us there is no downside and I can't imagine a time when there will be a downside for us. However, I will suggest a few things that could be downsides.
1. Buying into DVC at this time is not a good financial decision for you.
2. You don't go to WDW at least once every two years.
3. One of you (adults) is not a real WDW fan and has some reservations about DVC.
4. You aren't being realistic about the number of points you will need. In other words, you may be buying too few points and this could impact how you feel about DVC in the future.
Anything that could bring ******* into your family may be a downside.
One other point, you say you are on the fence. I'm not sure what factor(s) make you hesitate but I remember being on the fence and it took us several years to make our move. Many say that in hind sight they've looked back to when they were on the fence and wish they had jumped off on the DVC side sooner. I don't feel that way at all. I'm glad we waited, watched, and pondered because it gave us time to really think it through. If the circumstances were similar I would do it the same way the next time. Yes, it cost me more money but I also had my piece of mind.
Everyone approaches these things differently. I wish you well whatever decision you make.