Only because of the way Disney has set it up. Realize that there really is no difference between using a ticket of someone who isn't in the park to get a fast pass, and using a ticket of someone who is too small/young to ride or who doesn't want to ride, and who is in the park, to get a fast pass. The net effect on total rides/fast passes is the same. OP's original scenario is no different, from a ride capacity standpoint, then if his children were in park and were too short to ride or didn't want to ride. The only difference is that Disney has determined (as is their right) that the ticket media be scanned before it is allowed to be used to get fastpasses. Effect on number of fast passes / ride loads is same in either scenario.
I think there may be a way to get around this if you are local. Come to the park with your kids, have everyone enter the park, turn around and take your kids home (or have someone else take them home), two adults go back into the park. Voila! Incredibly cruel to the kids of course, but it would work I think.