For me, it's the mis-match. The ham doesn't work for me with the turkey gravy or stuffing - even if it was a very good ham, I'd want it on a separate plate with a different type of glaze or gravy. The pork at least fit in with the rest of the platter for me - it and the turkey and potatoes and stuffing could all share the same gravy (I admit the pot roast I tend to not mix as much with the gravy as it has its own flavor profile, but isn't offensive if the gravy or potatoes mix in with it).
I may love Thanksgiving turkey and gravy, and I may love buffalo chicken wings, but I wouldn't want the two served together on the same plate mixing their sauces and gravies. That's what the ham would do for me here - it just doesn't fit with all the other stuff on the plate. If it stays this way, it won't deter me from eating there again, but I'd likely just tell the server I can't eat ham and please leave it off the plate and I'll just enjoy the other stuff.
