I remember a poster who booked sixteen character meals in one week...to me, that's way too many. But to me, one is too many.
If you are concerned with the activity, alternate the character meals with something quieter.
I don't think it's true that all WDW restaurants except V&A are "geared" for families with children. The signature restaurants at the resorts are not geared that way. They have children's menus, but there is nothing specifically about the atmosphere that screams "family restaurant." They are full of families because they are at Disney World. Bistro de Paris is not geared that way. It doesn't even have a children's menu. However, plenty of families attend, with children of all ages.