We've always found good food at the CP and enjoy the atmosphere.
As to whether or not 11:50 is too early, that just depends on your family. Are you planning another large meal later in the day, or are you planning to just have counter-service or snacks later? If this is your one main meal for the day, you may want to make it a little later.
But not too late . . . you don't want to be eating lunch while everyone else is claiming parade spots for the 3:00 parade!!