I totally know where you are coming from. We used to live in Florida (many years ago) and we still have family and friends that live there.
We now live in Georgia and my DH works so many hours a week that my kids can go 6 days in the week before seeing him and then Sunday is our only time. Many Sundays he tries to spend time with us but also has to catch up on things that need to be done around the house or attend to other things that need his attention.
I try to carefully schedule our family time so that we can really spend quality time together. Sometimes, it's just a long weekend but mostly it's Florida trips because it's not far and we can drive there without killing our budget.
Long story short, we feel like we have to visit family and friends which totally cuts into "our" time. I know that's selfish but time time with "daddy" is precious.
So, our solution was to schedule time with friends and family as well. We explained this all to them and we try to see at least one family member and one friend during a week stay. If our stay is only a long weekend, we will have them meet us for dinner somewhere, such as DD or just explain that it's a long weekend and we will not be visiting this round. We also try to make an effort to attend other family gatherings and also invite family/friends up for visits from time to time. I think it helps us not feel so guilty and it's always good to see them outside of a vacation.
If you can't see them this visit, and if they are your family, they will understand as long as you do include them from time to time. If they are good friends and don't understand, then maybe you need to rethink that relationship. KWIM?
It's a hard situation because you don't want to hurt anyone and you do love them, yet, if you are like us, we really, really, need this time together! So start explaining this and make a tentative schedule. Let your Aunt know this! Tell her you would love to meet for lunch or dinner at DD instead!
Good luck!