I love these etiquette questions on the board, where all the "experts" make their decrees. Facing this question myself as the mother-of-the-groom, I went to a real expert Emily Post and here was her answer:
Q.
Is it necessary to invite all out-of-towners to the rehearsal dinner? We have so many I fear the rehearsal will be nearly as large as the wedding.
A.
Out-of-towners can be invited if your budget and the circumstances allow, but it's by no means a must. Traditionally the rehearsal dinner consists of the bride and groom, all attendants and their spouses or partners, the couple's immediate families and the clergyperson and his or her spouse.