It's been a LONG time since my own children were that young, but I know I would have been very embarassed if they were loud or misbehaving in what I considered an upscale restaurant. I would go, but maybe book for earlier in the day, when they are not sooo tired, and bring something new for them to play with if they get bored (something small and quiet). I could have never managed the mid-meal walk when mine were little, they would have taken that to mean we can get up and walk around over and over again.
We did Yachtsman for lunch when DS was five and it was fine. VERY QUIET! There was only one other family (also with a young son) sitting within our line of sight. All the children behaved very well and it was quiet and peaceful, a great midday break from the park. Have a great time!