Most of the doctors in the Family Practice I use are women. It's probably better to ask, in that case, if we should leave the male doctors for the males? At my kid's pediatrician office, all but one of the practice are women. So, same question there.
I think your question is skewed, probably toward your experiences.