Fast pass or not only matters to the first group riding the ride. The second group goes through the fast pass with a child swap regardless of how the first group rode the ride.
Assuming the 3.5 year old is too small to ride Soarin or Test Track, here's how it could work:
DH and big kid 1 get fastpasses for Soarin
DW and big kid 2 get fastpasses for Test Track (at a different time, clearly)
At Soarin FP time, everyone goes to the fast pass line, shows the little kid to the CM.
DW gets childswap pass and then goes and gets ice cream with little kid and bk2.
DH and bk1 ride Soarin
Later (whenever works for your schedule),
DW and both big kids get to ride Soarin
DH and little kid play in imagination
At Test Track fastpass time, everyone goes to the fast pass line, shows the little kid to the CM.
DH gets childswap pass and then goes and plays in the cars with little kid and bk1
DW and bk2 ride Test Track
Later (whenever works),
DH and both big kids ride Test Track
DW and little kid eat more ice cream.
Everyone (except little kid) gets to ride Soarin and Test Track, using only one of the tier one fast pass allotments. Little kid effectively doesn't wastes a tier one since no adult will have an available fastpass for her, but it's probably not that big of a deal. (You could, of course, substitute a Living with the Land fp if an adult wanted to ride that with the little one during Soarin time, but that depends on how old the bigger kids are and if they can ride something themselves.)
Hope this helps!