I think that "what ifs" matter way less than "what is", in a matter of speaking.
Family is very important if you are the sort of person who really depends on family for a social support network, and for whom family actually serves that function without complaint. I know a lot of people who do the family thing only because they feel obligated, and who grouse about family obligations all the time. If that's your actual family dynamic, my personal feeling is that you are better off without it.
My own extended family is spread across the globe. We like one another fine, but are independent sorts, and I in particular am considered an outlier for both my education & my politics. We all were very closely tied to one another when our parents were alive because they demanded duty visits on certain holidays, but once they passed we were quietly grateful to give up that hassle. DH grew up here, but his family is all gone now, so we could leave and probably would if we could find comparable jobs elsewhere. We certainly don't intend to stay here after retirement.