I am another that find non rsvpers very annoying. It wouldn't be so bad if not rsvping meant that they were not coming, but the first party I did for my DS, there were nonRSVPers who actually turned up. At my DS's party last year, there were 15 children invited and 6 did not reply, I had to chase the parents and all but one of them was coming and did come to the party, so no rsvp does not mean they are not coming.
A couple of years ago I did have one that did rsvp, the party was outside the home so I had paid for this child to come, paid for his lunch and for him to attend the party at the venue, then he didn't turn up on the day, no phone call, nothing. When I saw the mother at school the following Monday, I was not happy and she just said, oh my ex had him for the weekend, I forgot to tell him.

This same child was invited this last year to a party again and he did not rsvp, I called twice and sent an email and no reply. Unfortunately this child is now on our never invite again list. The child is quite sweet, but the parents unfortunately don't have any manners.
Anyway, I agree with everybody else, it is best to call the parents to see if they are coming, as there really isn't anything worse than the day before and finding out you have another one or two coming and have to scramble for favors etc. Some just genuinely forget, life is busy and hectic sometimes and they don't realise the rsvp date has passed, so giving them a call is the best thing to do.