Life's not fair. I won't take a table when I am with a group of adults or older kids, but will continue to do so when I am with my 4, 4 and 5-year-old. My daughter is special needs and cannot stand among crowds and wait. She will have a breakdown, which doesn't happen if she's at a table. If people want to take any extra seats at our table, by all means, go ahead.
Not an excuse - I just don't feel bad for doing it. You take your small child into line with you because you often travel alone with him. You don't have a choice, as you can't leave him at a table alone.
So unless I am going to get my stroller tagged as a wheelchair (which we can do) so I can bring it into the line, I am leaving my kids seated with my husband while I get their food. And as I said before, we rarely eat QS for this reason.