You could certainly just get FP's and then each member of your group rides 1 time with their FP. (Someone would have to wait outside the ride with the young child.)
However, baby swap need not interfere with your FP plan. Here is an example from our last vacation. We were at AK, DD(then 4) and I went to get FP's for EE while DH and 2DS's rode Dinosaur. (DD was not tall enough for EE.) We met up in Dinoland and toured for a bit. When it was time to use our FP's we headed over to EE and told the CM at the FP line that we needed to do a child swap. She gave me a little piece of paper. DH an the 2 boys went on EE using their FP's while DD and I found a nice little place to sit and watch the ride. When they returned, DH and DD went to look around the gift shop while the 2 boys and I went on EE using my FP and the child swap. Whole thing took less than 20min. and the boys got back to back rides while the standby line was 40 min.
Moral of my long winded story: You do not need to get the child swap at the same time you get FP's. One has nothing to do with the other. Also, using a child swap will allow the waiting adult to take someone (generally up to 3 people) on the ride with them and is a huge perk for older siblings.
HTH