I understand your point, but for some people it is a matter of balance, a vestibular issue. They feel disoriented if they cannot see clearly, and being disoriented in water often leads to panic. (I know that if you have done any lifeguarding you've seen the phenomenon of a panicked person swimming down while trying to get out of the water.)
Vision can be another issue. My DH and DS both are legally blind without glasses. They wear prescription goggles while swimming, and it makes a HUGE difference in how comfortable they feel in the water. Without goggles they simply refuse to get in, because they cannot gauge the distance to the side of the pool.
This is why they won't let kids at our Y start swim lessons with goggles. Once they can swim without them, they are free to use them. They just don't want kids to fall into the pool, and panic, because they don't have their goggles, and are wiping their faces instead of swimming.