We have been going for many years and we only do the parks 2-3 days. We rent a house with a pool and spa and my kids are happy to hang out by the pool on our off days. That being said, we try to do something every day. We go to Downtown Disney at least twice, usually once at night and once during the day. We go in all the shops and listen to/watch the outdoor entertainment. We take all sorts of pictures and get a snack. We never shop in the parks so this is fun and we let the kids pick out one thing to buy and we always let them get something at Goofey's Candy Company. On non-park days, we plan late afternoon dinners because that gets us out of the house on a "rest" day. I always have big plans to do something like mini-golf, picking oranges, heading to Blue Springs state park, take an air boat road... but then we get comfortable doing nothing and we never get there. Here is a thread with some more information:
http://www.disboards.com/showthread.php?t=2621999