Well I'll tell you we had a similiar situation in early summer. We booked a grand villa to HHI for June and invited both sets of grandparents along with our family of four. Everyone was on board and happy. My parents were really looking forward to it as they had been to HHI before and loved it. Well unfortunately two months before we were to go my father had a massive stroke and was to be in rehab for months so he and my mother could not travel.

Then my fil decided he really didn't want to go, so that left just my mil and our family of four for a GV!
We asked my bil who is a single dad of two kids if he would like to come with us, but he declined as he could not get off work. With all the stress of my dad's condition we didn't even think about our vacation until the month before when we knew he would be okay, but in need of lots of rehab. We toyed around with asking several others but decided it was also probably too late for them too to ask for vacation time and just decided to enjoy the GV for the 5 of us! It spoiled us immensely!!!
However in your situation if you think it would be fun to travel with the family member and child in your situation I say go for it, if it can work out for them it would be great for all of you! However if it doesn't enjoy the extra room!!!!