It shouldn't be any problem as long as you meet them and use your key in the elevator.
Last summer The Wife and I took a "family" vacation with my parents. They were on the 14th Floor of the Contemporary while we were in the Grand Floridian main building. We typically met up once a day for a drink or dinner and it was very easy to do. I, or my Father, would meet the visiting group in the lobby and use the room key to access the floor.
We all enjoyed being able to experience the atmosphere and view from the two hotels.