We did Crystal Palace for my daughter's 2nd birthday and the castle for her 3rd. Both places put confetti across the table and had a card signed by the characters present, and both brought out a little cupcake and made a very lovely fuss over her.
The difference to us was the character interaction. Obviously, your son probably doesn't care as much about princesses, but the seating area at the castle is smaller and the characters came around to the tables faster and spent more time with each table than the characters at Crystal Palace did. In fact, in a little over an hour at CP, we had yet to see two of the four characters at the meal. Eeyore and Tigger basically came up to our table, grabbed the autograph book, signed, and left, Piglet was in the other room the whole time, and we never saw Pooh. Also, "face" characters (ones that are allowed to talk/that don't have headgear on) can interact with the kids on a more personal level (our daughter had the princesses wishing her happy birthday, asking how old she was, complimenting her dress, etc) and sometimes are less scary, so that's something to keep in mind if your son is under 5.
We haven't done Chef Mickey's for a birthday, but love their character dinner.
Just our 2 cents.