The scenario you describe is NOT intended to be used with the Rider Swap pass.
There really is no way to "bridge"/"merge" two parties FP+ times together if they weren't reserved together in the first place and there are no other FP+ times available during that hour. Either you all buy your tickets together and get everything linked up ahead of time, or you are stuck with "making do as you go along", checking the wait times on standby vs. FP+ (sometimes it's worth it to standby, other rides are worth more to FP+ the line).
If all else fails, then you or your partner can decide to stay off a ride and hand G&G your MB so they can use the FP+ scheduled on it. I don't think that's against rules, since all of you needed a ticket to get into the Parks, and CM's seem pretty flexible about understanding that Adults are Interchangeable as long as the MB has the FP+ entitlement on it. They might roll their eyes, but seriously in the big picture of ride operations, as long as the FP+ is valid, there's not a massive change in quality/capacity of service should Granny Bertha choose to ride instead of Granny Martha.