Although I enjoy both places, I would say Crystal Palace for an Easter lunch. I just think the atmostphere in CP is much more Easter-y.....it's so bright and pretty inside, with all the glass windows....very spring! I can just see people singing, "In my Easter bonnet, with all the frills upon it..." Very Victorian!
And as previous posters have mentioned, the price itself is much better than the Castle for what you get. I always leave the Castle thinking I paid much more than what it was worth.
Just my two cents.....