I agree that there are as many germs on the door handle or handrails. But I can easily wash my hands and use antibac gel on my hands. Not so my kids' buns.
For me, it's also less about the disease on the toilet seats, and more about...how can I put this politely...the potential hard-to-see "leftovers" that grosses me out a little. So a quick swipe with the cloth just makes me feel more comfortable.