My husband and I have eaten at the Brown Derby and 50's Prime Time for Dinner, and at Sci Fi for Lunch
We've eaten at several of the more upscale Disney dining restaurants before and found the Brown Derby to be a very big let down. While the food was decent quality, I would not say it was any better than a restaurant similar to Chefs de France, or Coral Reef. While the waiters try to keep the ambiance upscale, the loud crowds, and fast service keep it from being fine dining.
That being said, we've had very pleasant experiences each of the three times we've been to the 50's Prime Time Cafe. The atmosphere is fun, and the food is good quality (I would suggest the shrimp cocktail, salmon and angel food cake).
We've been to Sci Fi for lunch which my husband loved, but I think it was really due to the atmosphere and the big screens with horror movies. I would say the food is similar to an Applebees or TGI Fridays.
If I could go to any again I would go to 50's...hands down!