We haven't been to the Brown Derby, but we went to Le Chefs during our last trip in January. It was good...not great, but good. The best thing about it was seeing Remy. It isn't worth a special trip when you're already going to be in the Studios where the Brown Derby is.
I have learned through my past WDW trips that:
1. No restaurant is so incredibly fabulous that a special trip must be made.
2. There are always good options in the area where you will already be touring.
3. My time and energy level are valuable at WDW, and I'd rather not spend them going to/from restaurants.
4. It cuts down on my family's frustration level if I can plan our meals around our park plans instead of jumping around so much.
Just my opinion, but this works for us.