If the 2-year-olds are capable of sitting very still, and using very quiet voices, the entire time you're there, it'd be OK. But that doesn't sound like a lot of fun for a two-year-old.
It's very small, very elegant, and very quiet. Even normal toddler voices would really stand out in there. And as others have said, the tables are quite close together, and there's lots of delicate fine china. I'd be a nervous wreck taking a pair of two year olds in there.
The tea will be a great experience when they're a little older. In the meantime, could someone in your party take the little ones for a walk around the GF grounds while you're having tea? There's plenty of places to run around and lots to see (ducks, bunnies, etc.). The kids would no doubt enjoy that a lot more than sitting in the tea room.