It is very interesting and here are a few potential reasons why a 'toff' might have had this viewpoint.
Hitler's popularity came partly from his destruction of the German Communist Party. Many people in Britain - not only aristos but middle-class folk too - were very much afraid of a communist revolution. Also, it seemed (until at least 1938) that Hitler had done a good job for Germany. Their experience of the Depression had been more severe than in Britain so when Hitler became Chancellor there were six million unemployed. Seemingly, he had put Germany back to work and restored national pride so the Nazis appeared to offer a successful model.
Of course there was also the lingering memory of the 1914-18 war. That had been terrible, wiping out entire generations of British men, so many were ready to accept Hitler's assurance that he had no desire for war, but wished merely to rectify the injustices of the Versailles Treaty...
And don't forget that the Duke of Windsor (aka Edward VIII) and his wife visited Germany after his abdication and expressed great admiration for what he saw there. His political sympathies were definitely pro-German (given his German background) and probably pro-Nazi. There are even claims he had an agreement with Hitler to be the puppet king of the United Kingdom if the Germans won the war...
And hereby endeth today's lecture...