I usually put a big stop sign around 4. BUT, that is when we're traveling from other places and all staying together. 4 is nice because we only need 1 room, don't need a suite, it's an even number for rides, if we split up, etc.
I've also gone solo and loved it.
Bigger groups: we're doing 6 for the Princess next year, but part of that is because we really want to stay in a cabin and not pay extra because of less people, and chances are we'll split up for some of the days.
When we lived down there, we'd have days that throughout the day, people would come & go, and we'd end up with over 10 of us all together. Since we lived there, we wouldn't be too stressed out about seeing everything, we'd make a bunch of fast passes for everyone since we were all linked, pick a headliner, something like HM, and a meet and greet. We'd spend the morning meeting a couple characters & watching shows, catch the parade, use our fp, then head to the GFlo for some wine and cheese boards, and come back for DEP/Wishes. There wasn't much that went into it because we were all pretty chill and people ducked in and out all day!