And how would a large group know that they are violating your rule about catching up to the rest of their party in line? And where do you draw the line, 3 people, 5 people, etc.?
When you are cutting in line (and that is exactly what you are doing), it is you who should be apologetic and okay if someone decides not to allow you to pass. After all, you are the one who is breaking societal rules. Even if it is to take your little one to the bathroom, or for whatever justified reason, you are asking to be treated differently than those around you. And that is okay as long as you are nice and appreciative. What rubs people the wrong way is when we make exceptions for folks and they are not appreciative or they are lying about what they are doing.
If everyone continually cut in front of others, the lines would soon be meaningless. So keep the cutting to a minimum. And if you really need to cut, be nice to those around you. It would make all the difference.
Disneyland will still be there if you have to wait 5 minutes for a FP runner to catch up to you.