To OP: Personally it wouldn't bother me, but if I were in your shoes I would leave as many people as possible in line (unless you need their help to carry drinks). It still wouldn't bother me with three people joining, but more than that gets iffy.
Now I don't mean to hijack the thread, but I see several people here saying that it doesn't bother them when an adult and child leave and reenter a line because you can assume the child needed to use the restroom. I'm curious what you all think about an adult without a child doing this? I often travel with an adult who needs quick access to the restroom due to a medical condition. When this comes up in a ride line she ends up leaving and skipping the ride to avoid conflict. I've considered seeing if this situation qualifies for
DAS on the next trip, but if it's considered acceptable to have her rejoin us in line then DAS would be unnecessary.