Hi again, OP! I'm the one who originally dismissed the Poly, but after reading your second post, I'm thinking in a different way. I have been to the Poly several times and have never found it to be loud and crowded. As a matter of fact, I think it's much more peaceful than the bustling BC, at least in my personal experiences. The GCH does get a little crazy at dinnertime, because the 2 restaurants are very close to eachother and they are very popular, hence lots of folks milling around waiting for tables. You can easily navigate around this by scheduling your TS meals, if you choose to have any, at "off" times, like as soon as the restaurant opens.
The GF health club is surprisingly close to the Poly. Your DH could use his pleasant walk along the beach to warm up and cool down. If he really doesn't want to walk, he take take the 3 minute monorail to get there and the 5 minute boat ride to get back (honestly, I just totally made up those times, but they're not too far fetched, I promise). Some of the disney resorts are so big that there are really rooms in the same resort that are further from eachother than the Hawaii building at the Poly is from GF.
Finally, at the Poly, you would have much more convenient, quick and easy food options, like CL lounge, Captain Cooks and room service.
I agree with SeaSpray about the Volcano pool slide. It goes thru the inside of the volcano and is sometimes completely dark when the red light inside isn't working. I mean black-out dark, no hint of light from either end of the tunnel. The quiet pool at the Poly is lovely and usually completely empty and steps away from Hawaii.