Palo is one of the things we think if as a nice experience but not worth a lot of effort for. I think we have ended up there on nearly every cruise, but after the first two times it was always because we wanted to dine with friends who are sailing but not dining with us in the main restaurant and having a night out together at Palos gives us the chance to all dine together the once. If it is ever just our family (or us sitting with all of our friends onboard) I would not bother. The only exception would be on a Med cruise if you could time it to see the sunset over Naples (that is how it was when we ate there in May and it was worth it for the view over dinner alone).
I am also the odd one out in that we did not care for the brunch at all. It was likely a bad day (my in laws and their friends were there too and we all came away with the same impression). The layout of the buffet items felt very crowded and busy so you lost the quiet ambiance, the food was mediocre at best and our entrees came out cold. Dinner is much better. Tea is fun and quaint if you just want to see the place (or just go for a drink at the bar).