Your question is pretty general, so without knowing what rides you want and who wants to go on what, it's hard to answer with specifics, but here are some general thoughts.
The 2 year olds won't have fast passes, since they don't have tickets. They won't need them as they're basically "along for the ride" when entering a fast pass line.
A babyswap pass is good for up to 3 people to come back later and get in the FP line.
How the first group of adults ride the ride is irrelevant to getting the babyswap - so the first group can use a FP+ and get in the FP lane or they can get into standby. Either way, you can get a babyswap assuming you have a too-small child.
For most rides, they'll give the babyswap pass to the waiting adult at the entrance, but they'll usually wait until the rider is actually in line. Some CMs will give the pass to the rider, not the waiter. And on some rides (BTM for sure), they'll give the rider a special card to turn in at the actual ride to receive the fastpass. The general goal is that the first group has to actually ride when you get a babyswap.
Given that you have two too-small kids and a ton of adults, you could - theoretically - get two baby swaps for each ride as long as you didn't do anything to indicate you were all together. I'll leave it to other Disers to determine if that makes you a horrible person or not. I will just say that, having travelled with a similar group, we really never found the need to use the kids for multiple babyswaps - we were plenty busy just trying to keep up with what we had!
So, here's an example:
3 adults get a fast pass to Test Track (group A)
2 adults get a fast pass to Soarin (group B)
One adult from Group A, one adult from Group B, and one child approach the fastpass to Test Track as a group and ask for a baby swap pass.
[If you decide to go for it] a second adult from group A, a second adult from group B, and the second child approach the fastpass to Test Track as a group and ask for baby swap pass.
Group A rides Test Track using fastpass.
Group B and the kids go ride Finding Nemo.
Later,
One adult from Group A, one adult from Group B, and one child approach the fastpass to Soarin as a group and ask for a baby swap pass.
[If you decide to go for it] a second adult from group A, a second adult from group B, and the second child approach the fastpass to Soarin as a group and ask for baby swap pass.
Group B rides Soarin using fastpass
Group A and kids go get ice cream.
Later,
Group B rides Test Track using the babyswap while Group A and the kids ride Nemo again.
Later,
Group A rides Soarin using babyswap while Group B and the kids ride Nemo (yet again)
Later, the three people you like best ride Test Track again using the second babyswap [if you did that]
while you stick the other two people with Finding Nemo for the 20th time.
Finally, the three people you like best ride Soarin again using the second babyswap [if you did that] while the other two poke their eyes out having heard Dory search for Nemo one too many times.
Hope that helps!