We were also very anti-video game in our house, after watching my nephew the "gamer" grow up with a DS attached to him at all times. BUT, after visiting a friend and playing some of the sports games on the Wii, we were hooked.
There are so many games that are so much more than the typical video games, which I hate. We really do enjoy golf, tennis, and bowling as a family. We got ours when my son was 5 and that seemed to be a good age - my younger niece came over and became frustrated pretty quickly.