As for kids in the nicer restaurants, you and your friend know your kids. Our girls have been to has been to California Grill, BOG, Narcoosees, Citricos, Artist Point and Flying Fish, some of them a few times (CG is DH's favorite). Our ODD, now 6, was fine at all of them since we first took her to CG at 3 - she loves food and they still bring kids menus and crayons, so she actually stayed in her seat, ate, and did great. Our YDD was more of a challenge at 21 months on this last trip, because she didn't want a high chair and didn't really stay in her seat. That said, our last time at CG she stayed on a chair most of the time, eating, sat in my lap some of the time, and lay down to have a cry for a few minutes by the window behind my seat. But with the ambient noise you really couldn't hear her and she was fine soon after, and she wasn't in anyone's way. At Flying Fish she was in a booth next to me, and DH and I took turns taking her outside for walks after she was done eating. Again, moderately annoying for us, but she did not disturb the other guests at all. And your kids are older, so if they would be interested, it might be nice to expose them to some better and different food, especially if you're using DP credits. It really depends on your kids and whether they'd enjoy it and whether you'd enjoy it with them along even if they didn't enjoy it, if that makes sense. WDW is one of the few places you can go to nice restaurants with good food and bring kids and not have other patrons or staff give you the sideeye about it, and that's one of the reasons we like going.
We've been doing nice restaurants with our kids since they were little - it was my youngest who was the sixteen year old at Jiko, and Jiko is the only one at Disney were we felt that our kids didn't belong. It wasn't that they weren't well behaved (this was back when they were probably six and seven), but that they were some of the only kids in the restaurant. We also had a server that didn't enjoy serving kids (one of the very few bad servers we've had at Disney). We didn't go back with them until they were both tweens - and this last trip - just a year ago - we noticed that Jiko was still mostly adults with very few families. That's different than what we've had at California Grill (lots of families) or Narcooses (fewer with small kids than CG, but still comfortable) or the old Flying Fish (haven't been to the new one - its much more adult in its decor though). Some people are sort of oblivious to "we have the only kids in this restaurant" but I get an uncomfortable feeling when that's the case.
So while I agree its about the kids, its also about you and how you feel about the kid ratio.