I would think that as long as your boys can sit quietly for the length of the tea that you would be fine. If, however, you think they would get bored with it and want to run around and be boisterous I might rethink going. The Garden View Lounge is a bit "upscale" (as is the GF in general) and lots of people go there for a relaxing afternoon. As long as your boys could handle a quiet hour or so in a nice restaurant I'd say go for it.