I would recommend taking the bus from Magic Kingdom to Saratoga Springs, then once you get over there get off at Congress Park and use the walkway to Downtown Disney.
If you were to walk over to the Contemporary and use their bus it will also stop to pick up guests at Polynesian, so I don't think going there will be a lot faster.