Sorry, buses don't (generally) go from resort to resort. The best way I can think of to get to Poly from AoA is to bus to MK, then get on the boat to Poly. You can also ride the resort monorail to Poly, but I think in this case the boat would be quicker.
Returning to AoA, just reverse the process, except in this case the monorail from Poly to MK would probably be faster than the boat.
any idea on how long a trip like that would be? i wish there were cabs! i'd gladly pay for a 10 minute cab ride vs a free hour long "tour of the property."
I'd budget 30-45 minutes for the trip, which includes waiting for the bus.
There are definitely cabs. I'm not sure if you'd need to call a cab to AoA or if they're waiting outside already. Lobby CMs can call one for you if need be. Driving yourself or riding a cab is ALWAYS the fastest way between two resorts (unless they are walking distance apart like some resorts).
Bus to Magic Kingdom, boat or monorail to Polynesian. I'd say more like 60-90 minutes to go between resorts. Do not use Downtown Disney as a bus hub unless you absolutely have to. And yes, you can get a cab. Go to the front door of the resort and someone will call one for you if there isn't one there already.