The summer after DH and I both turned 40, I booked us for 5 nights at Peter Island Resort - it's in the British Virgin Islands. I had never been away from my kids for that long before, and never flown away from them, so this was huge. I paid a friend (not much, but I felt I needed to) to stay with them.
Peter Island is very quiet, hardly any kids (they don't prohibit them, but there is no supervision for them, and the rooms only have one bed), and relaxing. It was so fun. You fly through Puerto Rico to Tortola, then they have a ferry boat that takes you to Peter Island. I would love to go back someday!
We have been to WDW a few times before we had kids, so there I'd always bring them (while I can!). If it's adult only, I'd pick somewhere that you wouldn't take kids. Like this past weekend, I went with a girlfriend for her 50th birthday to a small town nearby to shop and eat and relax, no kids (or husbands) and it was wonderful. Kids would have been bored.
Have fun planning!