Originally we were going to invite about 65 people. This included immediate family on both sides (aunts, uncles, first cousins, etc) and very close family friends.
We figured out what we could afford vs. what elements we really wanted to have (ie: a dessert party, rehearsal dinner, etc). Before we sent out our save the dates, we cut 20 people from the list in order to have what we wanted. We cut my dad's family, mainly because I am not super close with them, and because one of his sisters blatantly told people around town that she would not go, and the other would not have gone because of the money, so we just cut them & their children and that took care of thirteen people off the bat. We also cut an additional 10 people who were relatives, but not super close to us.
We sent out the save the dates to 40 people. A few months later we sent out newsletters with pre-rsvp's because we needed to have a good idea for budgeting reasons. Three people said no at that point, leaving us at 39 people including us.
Right after the invites went out, DH's family completely backed out, bringing our numbers down to 32 including us.
I will say that I am really glad we cut our numbers. Weddings are so full of drama, and I am so grateful that the people we invited, albeit a small number of people, were family members and/or friends that truly cared about us and wanted to be with us on our wedding day. Had we invited the other family members (who, like you mentioned, are people we see once or twice a year) we would have had to cut really important elements from our day and to me it just wasn't worth it. I'd rather have a really great experience with less people than have to invite "extra" people just because I'm expected to. We did deal with some backlash from our decision, but I just stopped caring. It's your wedding day, so don't let anyone push you into inviting people if you don't want to. Just my 2 cents!
PS - we live in CT, so travel was a major factor for all of our guests. 29 guests came down from CT, and 3 came from TN.