I'd go for it! I've never gone to WDW without kids but I imagine it would be easier to travel and move about. We spend a lot of time walking and stopping at bathrooms-you wouldn't do that with kids.
What I would do if it were my husband and me is go to Epcot on Day 1. We've always wanted to "drink around the world" and can't on our family trips. There's a lot of things in Epcot that we move through quickly with the kids. It would be nice to take our time there.
On day 2 we'd go first thing to HS to ride TSM and Star Wars-our favorites- then back to the resort for relaxing pool time and a nap.

Then we'd go back to MK that night for late night rides and fireworks! If you're not huge Star Wars fans like us then skip the park hopper and go to MK in the a.m. and the p.m.
The 3rd day we'd probably have a nice breakfast then either relax at the pool or go to DTD to shop around and have a nice lunch.
You can get a lot in and relax a little too without kids.
Wow, now I want to go with just hubby for a long weekend!