Well, a New Englander by birth, I vote for Cape May. I'm one of the ones who has consistently had nice meals there. We love clams and mussels, shrimp and fish. Although small, the clams are nice, and the mussels are as well.
We like the chicken on the buffet too. It's a meal that can please many people, so we take our vastly different "tastes" family there when we are all together.
I'm absolutely not going to eat at 50's PT, but that is my personal matter. I do NOT like the interaction there, although folks do love it.
So for me, no contest. Cape May all the way!