Your family sounds a lot like mine. We go to Disney World once a year and love the Poly...it's our favorite, too! After 5 trips staying in Deluxe resorts (2 of which were at the Poly), we finally became DVC members.
If you vacation at Disney World every year and are staying in the Deluxe resorts, you should definitely consider DVC. However, like the other posters mentioned, if you love the Poly and ONLY want to stay at the Poly, DVC would not be a good investment. The point costs there are structured so that it would likely cost you more than just paying the Disney hotel rate.
We were spending about $2000 for a week at the Poly (and similar cost for the other Deluxes). When we realized we could buy into DVC and get 50 years of vacations for what it would cost for 5 years at the Poly, we were OK with "sacrificing" our stays at the Poly! We recently bought AKV (we have stayed at AKL before and enjoyed it), and have absolutely no regrets. It's truly not a sacrifice at all, because we still get to stay in the Deluxes, and now we have the option of getting larger accommodations if we need them.
Here is something you might consider if AKV appeals to you, but you are not willing to give up the Poly completely. Figure out how many points it will take to stay at AKV for the season and length of time you vacation. Divide that number in half, and buy a small contract at AKV for that many points. With banking and/or borrowing, you can use your points to stay at AKV (or any DVC resort) every other year, and on the alternate years, you can still stay at the Poly (paying out of pocket, of course).
You may find that you really enjoy the savings and options DVC has to offer, and you can always add on more points in the future.
Hope this helps! If you have any other questions, feel free to post or PM me.