It really depends on the size of your family, the games you expect to play, etc.
I have four remotes and two nunchucks. However, we very rarely use the second nunchuck. And about the only time we use a nunchuk at all is for Mario Galaxy and Zelda: Twilight Princess.
Mostly we play the Sports Pack (Bowling, etc.) or Boom Blox which is an incredibly addicting game for all ages!! I highly recommend it.
Back to the remotes - if you don't want to spend $120 for three extra remotes right off the bat, get one extra and see how it goes. You can always add more controllers, remotes or nunchucks as the need arises.