I would say, yes, go ahead and take them . . .
BUT, be ready to get them out of there if they start to create any kind of disturbance for the other diners. This is an upscale place where patrons have a right to expect a certain atmosphere. A well behaved, quiet child coloring or eating, etc. doesn't take away from that. A child who is screaming (or just talking loubdly), throwing things, etc. does take away from that.
My husband and I will be eating at Blue Zoo in Feb. We happen to be leaving our 3 year old back at the AKL with my mom that night so we can have a night out. (We're heading to Jellyrolls after dinner!) However, I won't be upset if there are other quiet kids eating there. I will be upset with their parents, though, if they don't cart them outta there at the first sign of them making a scene!
We have, at Disney and other places, taken our little guy to lots of nicer restaurants. Many times, it's worked out great. Other times, we've ended up either boxing our diner up to go or having one person eat while the other took the wee one out for a walk. We never let him ruin the dining experience for others. That's one of the down sides of being a parent.