We've stayed at POR 3 times, and been in 3 different mansion buildings. What we've discovered is that your location within a building is more important than the building. We've stayed at:
1. Parterre Place - our room was on the far side facing the parking lot. It was about as long a walk to anything as you can get in the mansions.
2. Acadian House - we were on the inside of the courtyard. It was very convenient to the North Depot to catch a bus. Probably the same distance to the main building as the Parterre Place room, but a nice walk on Ol' Man Island. This makes the pool a nice walk as well.
3. Magnolia Terrace - Another courtyard room, but at MT the courtyard faces the river. About the same as Parterre to the bus, a little shorter (but still across the Island) to the Main Building. Can't get closer to the pool!
We tend to prefer Acadian and Magnolia, since the pool is important to the kids. We also like the walk past the pool in the morning for breakfast. I can't speak with any knowledge about the Bayou buildings.