RSVP is often accompanied by a date, which tells the person being invited that a response of attendance or not is expected by the given date. To not respond would immediately remove that person from many guests lists...at least among people I know.
If someone cares enough to send you an invitation to a party (and I mean a true party, not one of those pretend parties that are really sales forums) then you should really have the decency to respond.
Having to chase after a response is equally improper.
Start shortening your guest lists by these non-responsive people and at least save yourself some stress.