It's possible, but it is hard I think. We've made it a little more relaxing by staying in deluxe resorts. The pools are nicer and with a lot of them you can get a balcony or patio to relax on. We like to stay where we can walk to one or more parks (BC, YC, CR), which takes some stress off of having to use buses.
On our last trip, we split between CR and BC. I found that staying at CR and being able to walk to MK instead of spending 45 min on a bus really helps. We had a first floor patio room and I enjoyed sitting outside at night or in the morning. At BC, I requested and got a full size balcony. We were overlooking the woods, so it was very peaceful. Being able to walk to Epcot is also wonderful.
I think going for longer trips makes it more possible to relax. If you are only going for 4 days or so, it's hard to fit in downtime because you want to get in as much park time as you can. We try to go for 8 nights, and even then it's hard to find a lot of downtime.