The textbook, perfect, PC answer is that DH and DD go, you watch the other kids, and then you and DD go later (and she gets to ride twice... the oldest/bravest kids always come out ahead, haha).
If you really have no desire to ride it, then yes, you can use the rider swap pass for DH and DD to both go a 2nd time. You would not be the first family to do this... that day, even.
Some people are WAY too uptight and self-righteous when it comes to Disney. Believe me, you're at such a tremendous disadvantage (as far as waits and efficiency) when touring Disney with small children, that working the system for little perks like this wouldn't even begin to make up for the time you lose in other areas (e.g. using rider swap on something like Tower of Terror takes it from a 20ish minute experience w/ FP to almost an hour). So I wouldn't feel bad at all about using it.
It's not like DH going twice w/ DD costs anyone waiting in standby to have to wait any longer than if you went that 2nd time. You "earned" the rider swap pass (and paid $$$$$ to take your family to Disney in the first place)--no one else should care how you go about using it.