It's possible you would have felt that way.
But I felt, when I had those feelings, not just sad, but also happy. I was happy to be experiencing it, and I did *different* things than what we do. I saw "it's tough to be a bug", to preview it for DS. By doing that, I know we don't have to take the time to see it with him for a LONG time, and didn't waste any family trip time in doing it.
I find little things that I can tell them about, and then it enhances the family visits.
When I see kids having fun, I get sad, but know that DS and DH are at home having fun dad/son time. When I see kids tantruming, I think "I'm glad that's not me having to deal with it" AND "aw now see that's not so bad, I'll try to deal with tantrums at the parks better next family trip".
I wouldn't go to WDW or DLR with only my husband, but I will go solo. Then again, utter lack of competent grandparents kind of negates even the possibility of a trip like that happening!