DH and I will be married 25 years in Nov. We talk every day about a variety of things. It could be something minor, like traffic (DH has an hr+ commute each day or what happened at work or something deeper like politics. While we know where each other stands on certain issues, new twists on issues comes up regularly.
When our dds were younger (teens now) we worked opposite shifts and it took a toll on our marriage. It was rough for a couple of years but we pulled through. One thing that was tough was not having enough time with just the two of us where we could talk and reconnect.
I am a talker and Dh is more reserved. But over our marriage we have learned to read each other so sometimes talk isn't necessary when you can read the look you are getting.
OP, to be blunt, when I read your post that you avoid going away with just DH when he suggests it, I thought you may have problems down the road. Since DH suggested it and you said no, it seems like he wants the situation to change.
To say you don't want to spend vacation time alone with your DH seems like a red flag to me.