We stumbled across Columbia Harbor House one visit and had a fish and chicken basket which was quite tasty. There is an upstairs dining area, which was roped off when we were there, but we asked a CM if we could eat up there and she said, "Of course" and took the rope down. The upstairs area was quiet and we had a table by the window looking out over the walkway and enjoyed people watching and having a birds-eye view of the surroundings. Once the rope was down, a few other guests came upstairs to eat but it never got crowded.
Last year we did the Keys to the Kingdom behind the scenes tour, and they served us lunch upstairs at CHH. It's a nice place to relax away from the crowds.