My extended family (sisters, brother, in-laws, teenage and early 20s nieces/nephews) went to Cape May last fall. We loved it!
A quick caveat. As much as I personally enjoyed it, it's not a 'must' for me simply because as good as the food was, I can get good seafood throughout the year. I grew up in Louisiana (and visit frequently, going to the Gulf coast at least once per year) and live near the coast in the Mid-Atlantic region (and can get really good seafood here as well). Given that, when I'm WDW on trips that usually last 5 to 6 nights, I prefer to spend my dining money/time at places that are more unique to me personally.
Our experience: the crab legs were plentiful, flavorful, juicy, meaty, and perfectly cooked (not cold at all). Everything else on the buffet was excellent (seafood and non-seafood options). Dessert was fantastic. Service was really good. We were seated not long after our ADR time, on a busy weekend, and had a fairly large party. It was a good experience. We left STUFFED.
My entire family (all who are able to get really good seafood regularly) loved the meal.
We're planning another large family trip this fall, and are staying at BC. We don't have an ADR for Cape May (because we want to try a few new places and are able to get good seafood at home). However, we have a few free nights without ADRs and given as we'll be at BC, some of us have said that if Cape May can accommodate us, we'll definitely go if able.