I don't think you are crazy at all! I have debated buying into DVC for about 5 years now and still don't own. We too visit WDW frequently and mostly stay in deluxe hotels (GF and WL are our favorites, with BW and Poly close behidn).
We stayed in a 2-bedroom Savannah View at Kidani with extended family on our last trip on rented points. It was a great experience and I would do it again. However, it did not replace my favorites.
One thing I have learned through a LOT of research (5+ years worth) is that I personally enjoy "hotel"-style accommodations a bit better. I like the hustle and bustle of the main wilderness lodge, I like all the counter and sit-down restaurants of the Grand, Poly and Wilderness Lodge, and I especially like the fact that the regular hotel rooms are refurbished a bit more often than the DVC rooms are. And unfortunately, though I love WL, I really dislike the WLV decor (same with BWV). This is all a personal vacation preference and not a criticism of DVC. DVC is a great program for those who it is a good fit for. If our little family ends up blossoming to a family of 5, we would probably buy for sure, despite the decor. We know we want to vacation there for many years and we know we can't afford the Grand or Poly every time (for the 5th person daybed).
Even now we are considering it again, due to our stay at AKV Kidani. The rooms are really decorated to our taste, the pool area is now my 2nd favorite on property (the GF cabana's are still my top fave) and with AKV, you have an opportunity to be in the main lodge. But with just the 3 of us right now, we're content with regular hotel rooms. I feel less guilty about ordering room service, and really enjoy that someone else makes my bed every day for a week!
But maybe perhaps just a few points.... to be able to go AKV every other year.... (see, the debate is never ending!)