We had to cancel brunch. We called the night before, and it was no trouble.
We had just done dinner at Palo on back-to-back nights (our reservations with friends, and friends reservations with us) and were so stuffed, there was no way we could eat there the next morning too.
Just give them as much advance notice as possible, and you should be fine.
Enjoy! It's wonderful.
Scott