I think this is one of those mileage may vary type things. The first CM we ever asked about this was very specific that my younger son also needed to be at least 7 years old. This was just when the boys wanted to ride together in a Buzz car while DH and I rode along in the one behind.
When my younger DS first turned 7 and wanted to ride certain things without us and just with his brother, their own Buzz car, Heimlich's choo-choo, and Gadget's go-coaster for instance, the CMs would sometimes ask him his age and sometimes not. The year he was 8, he was asked his age almost every time before he boarded.
Nowadays, DS9 and DS11 ride a couple of rides together, just the two of them (I like to do this with rides I can mostly see from beginning of queue to end of ride since they are still so young), while DH and I watch and wait. Even these days, DS9, who is small sized, still often gets asked his age by the ride CM before he is allowed to board with his brother.
So either they have gotten stricter and more consistent about the policy in general, or we have just run across some very strict CMs.
Anyway, OP you should be able to put the two older kids in their own Buzz car and let them ride certain things alone together if that helps you any.