No, I don't think that is the wrong way to go. Start with something to cover at least your main trip and then go from there. You can always add on later if you find you can't make what you have work.
When we bought, we bought enough to cover our summer trip. We knew that eventually we would want more and have since added on but figured it was good to start slow and make sure we were happy with
DVC--and we are!!!
So, if you know that one of your trips would be 150 points, I might start out with 175 or so, just to give flexibility and go from there. With banking and borrowing, it is possible to get a 2nd trip in maybe every few years while you wait to add on.
Good luck!