If they are family and you are reasonably close, you will probably love seeing them. I have a similar situation when I go to Disney. I have relatives with a (rental/vacation) house in Reunion. They know I go to Disney a couple of times a year. They want me to spend time with them when they are there. I've never slept in their home but I'm constantly invited.
My brother understands completely. (His daughter owns the home.) He says to me that he would prefer to be in the center of things at Disney when he comes, even just at a resort for the restaurants and the ability to stroll around. But that never happens for them now. Also this is an 8 bedroom vacation home, so they don't live there but use it for vacationing. I don't golf and swimming means dangling my feet in the water and sitting by the pool, so there is no added attraction other than spending time with family that I don't see often. The home is on the rental market when they are not there.