We usually go for dinner on our arrival night. We have quite a long flight and generally don't get in until later in the afternoon, and we're all pretty tired. We usually grab dinner at Raglan Road, do a quick perusal of the World of Disney, and head back to the hotel so we can get to bed early. We are rope droppers, so I like to try to get a good night's sleep in.
That said, I could totally spend a week in the parks and never take a break for DD. We feel like the reason we go is to spend time at the parks. There are swimming pools and mini golf just about anywhere you vacation, but only Disney parks have the attractions, characters and shows we love! So if there isn't a huge draw, I'd say skip it. We actually might do that this time and just do a resort meal on our arrival night instead!