The reason most dentists recommend having the pulled even if you are not experiencing any current problems is that you probably will eventually. When that time comes, it may be much worse than if you had them removed now.
Since the are so far back, most people do not brush them as well as they should. Our dentist told my daughter that they tend to decay easily and people often end up with an abscess.
To tell you the truth, having wisdom teeth out now days, is a piece of cake. My daughter had her wisdom teeth out about a year ago. The procedure took about ten minutes. They gave her the twilight sedation which did not make her sick. In fact, she felt fine after about a half hour of waking. She had to eat soft foods for a couple of days, but had no problems.
A lot of people used to get dry sockets after having their wisdom teeth removed. Often, that was the result of vomiting after the sedation. The stuff they used to give made many people ill. The force of the vomiting dislodged the scab area and revealed the nerve. Having a dry socket is much rarer now.
On the other hand, needing a root canal is not a great deal of fun. If your wisdom teeth end up abscessing in the future, that is a good possibility. OP, if I were you, I'd have them removed.
By the way, I had mine out when I was 19 and have never regretted it.