I remember my youngest one wearing a lifevest when she was 3 years old, she hated it and that was the last time she ever wore one. She was more comfortable learning how to dive and swim than wearing a lifevest. I let her go for it ( while I was in the water with her of course ) as long as she could touch the floor with her feet.
She's 7 now and she swims like a fish , goes into the deep part of the pool without a problem but she's not allowed to go there without an adult.
There was never an in between situation with her and I think it all has to do with older kids always around her , and she had to catch up with them to keep up with the fun, along with her not having any fear.
She did have swimming lessons this year in school , one of the days I went to watch them and the swimming teacher basically told me that the other kids were holding her back and asked if it was ok with me to have another teacher take her to the deep side and teach her other things. I gave them my permission.