It's your call. If you call and cancel you could risk losing your preferred time and restaurant, but if you modify the number of people in your party, it could open up a spot for someone looking for a reservation for a party of two. You could call and see if theres availabilty, and if there is book it. Then you could cancel the other reservation without worrying about losing your preferred time.
Hope you have a wonderful trip!