This. The only way management will know is if people speak up about it. When people don't say anything, they have no way of knowing. Yes, they watch, but without you saying either after brunch or at check-in for dinner that you felt the service was insufficient, the manager might have assumed you asked to be left alone as some people do.
ETA: Also, brunch and dinner on the same day?? WOW! That is a LOT of food!!