Just my opinion of course, but I would write it off as lesson learned. Your Doctor is human and it sounds like maybe you need to find a new Doctor as you have lost trust in this one.
My father went to see his Dr. several times over the course of 4 weeks with undiagnosed pain. Severe pain - he missed work and he never missed work. Week 4 the Dr. wanted to send him to a psychiatrist as he thought it was all "in his head" (he was depressed) but instead my Mother brought him to the E.R. - he had a C.T. scan and had an aortic aneurysm that was "leaking" - good thing it was leaking because it should have killed him weeks before. He had immediate surgury, difficult recovery, several life long post surgical complications. Wouldn't ever think of suing, the way he figured it, how could the Dr. know what it was, when it was the type of thing that should have killed him in hours not cause him pain for weeks.
Several years past, and he passed out at home - turned grey, Parametics couldn't find a pulse. Turns out, with the medicines he was prescribed he was missing two important supplements - Potasium and Magnesium. He went a few years depleting his body of needed minerals, a KNOWN side effect of the drugs he was prescribed due to surgical complications. The Doctor "should have known" - the Pharamist "should have known" who knew? He survived again after several weeks in ICU (for which he had to pay several thousands of dollars out of pocket, again) - but mistakes happen, again, he would hear of suing.
There was a third incident but this is getting too long.
And the reason my father would never consider suing, even after 3 near death experiences two of which were completely preventable and one which the misdiagnosis caused him 4 weeks of incredible paid, after some 10-12 weeks in ICU (when you add all 3 incidents together) after lifelong painful complications, was that he felt that as humans, people make mistakes, and he was an incredibly forgiving man. He figured it didn't kill him, and frankly, he had strong opinions about frivilous lawsuits driving up the prices of Health Care for everyone else, that he felt was a bigger wrong.
The first time Dad was in the hospital for an extended period of time, he was in a shared room in the Cardiac care unit - the man in the next bed was an uninsured farmer, and he and his wife had several tearful meetings with realtors because they had to see if they sold their farm property they could come up with enough money for him to have the bipass operation he needed. It was heartbreaking and something my Dad could never get past.