No! We have stayed at POR many times. For a while they were putting us in Alligator Bayou, then they started putting us in the mansions. We actually like the Magnolia Mansion the best, it's a short walk over a bridge and past the main pool to the lobby/food court.
You can certainly call Disney and put in a request if you would like a particular building/area. I have found that they usually ignore my requests, but at check-in, if I ask for a specific area, they are always able to accommodate me (note we typically go during slow periods).