A party of 6 vs a party of 5 will still be seated at a 6 top table.
A party of 8 vs a party of 4 requires a different table top size. OP, you will be allowed to be seated with only 4 on an 8 reservation but you may have to wait some time as they will not seat you at the 8 top with only 4 of you dining. Depending on the CM and how cooperative they are, combined with the table layouts in whatever restaurant you are doing, you may be seated quickly or you may have a wait.
You cannot call and modify an 8 person reservation to a 4 person. That would require a cancellation and rebooking, and if there are no 4 tops available, you will lose your current reservation. So I'd keep the 8 top and be prepared to wait while they juggle tables in an attempt to accomodate your smaller party.