I just went to my eye doctor last week, and he told me some truly disgusting stories about patients getting really nasty amoebic eye infections from wearing contacts swimming.
*** warning: skip this part if you're squeamish ***
Basically, "critters" in the water can get stuck to the outside of the contact. You put the contacts in their case for the night, which lets the critter crawl onto the inside part of the lens. Next morning, you put the lens back in, thus sealing him onto your eyeball --> where he can then burrow into your eye! *barf!!!!!*
*** gross part over, safe to resume ***
I've ordered a box of one-day lenses so I won't have to worry about it. I'm sure the actual risk/probability is very small, but my eyesight is horrible already, and the very idea grosses me out to the point where I'm willing to spend the ~ $15.