I have a problem with saving them and coming back.
They are to be used so that both parents can ride as close to the same time as possible so they don't have to wait in the same line twice. When you switched at the exit you road right then and there. I have no problem with the older siblings going then to. But should be used then.

Huh? Where in the world did you get that idea?
Think through how the swap passes are actually used. Say you have a family, mom, dad, Big Kid, Little Kid. Mom, dad, and Big all want to ride Tower of Terror. Little is, obviously, little. So they get a baby swap pass. Dad and Big get in line for tower of terror (maybe fastpass line, maybe not).
In your world, mom and little now have to sit on a bench in front of tower of terror for the next half hour (at best, assuming fast pass) twiddling their thumbs waiting for Dad and Big so that mom can ride immediately after they get off. Mom and little are bored and getting more bitter at Dad and Big with each passing minute. They aren't having fun and aren't spending money. Why in the world would you subject them to that????
Instead, the reasonable (and Disney-sanctioned) thought is that Mom and Little don't sit there waiting for Dad and big to return - isntead, they go have some fun for themselves! When Dad and Big get off the ride, mom and little may be half way through Disney, Jr. or Beauty and the Beast show, or maybe they just bought an ice cream, or maybe they're watching Little participate in the Jedi show. Whatever... they are off having their own fun and will meet up with Dad and Big when they can. And mom and Big will go back to Tower of Terror later in the day while Dad and Little go do some fun stuff on their own.
The swap is intended not just so that the parents don't have to miss out on a ride, but so that the little kids can actually enjoy themselves in the park as well! It is, after all, a vacation for the whole family!