We've seen children of all ages at YS--usually very well behaved! I'm sure some parents have chosen to remove their little ones when their behavior warrents it.
Keep in mind, though, that they do not make accommodations for children (I think they have high chairs, but not sure about booster seats) in terms of things to entertain them. You'll need to bring some quiet toys/books for them, if needed.
It's a nice environment. Fairly quiet (we tend to dine a bit early or late, rarely at the prime dinner hour), not at all stuffy, with good food and service.