I do agree with this in general. I love brunch at Palo. But I feel like pushing the time to make it rushed and shorter is wasting the brunch experience.
And given that the OP used the phrase "feel like I'm wasting a day" I am more inclined to say don't do it if it is going to cause you stress and you feel like it is a waste of time. That said, there is no way you can do it all - even on the longer cruises. You need to pick and choose.
Honestly if you are going to be rushing, watching the clock, and feeling like you are wasting time, I wouldn't bother with it.