Hi BelleNJ-
We've never been to WDW without the kids, but I've dreamed about it!! (DH isn't the Disney fan I am and can't imagine going there without the kids). We've been 3 times since we had kids. The first, our DD was 9 months, the 2nd, she was 16 months, and last month, she was 2 and we took the new baby for her 1st trip at 6 months. While I immensely enjoy taking them and spending time together as a family, I would love to be able to do some adult things. I'd like to eat at V&A, ride all the thrill rides, take some of the "behind the scenes" tours, sit by the pool with a good book, take time to slowly enjoy all the things Epcot has to offer, etc.
We haven't been on a "mom and dad" vacation since our 2nd was born; however, we went away for 4 days when DD#1 was 5 months and then another 4 day trip when she was 11 months. We did miss her and I called at least once a day to check on her and say hi. Despite the fact that I missed her, it was REALLY nice to have some alone time with DH. Like you said, "sitiing down for dinner", is something you lose when you have little ones. We enjoyed taking a nap every day, staying out late, nice dinners, quiet time by the pool, etc. Making time for those things is critical to the health of your marriage and the health of your marriage is key to your DDs happiness. So, stop stressing, and go have fun!!