Personally I would not let this ruin my trip. If it were me I would tell them I have ADR's here and here. If you would like to join us at dinner I can change my plans on such and such day and make ADR's for the group. That is the end of the obligation. If they are eating a Turkey leg for dinner...