WDW restaurants are not open continuously from breakfast through dinner, even if they serve all three meals. Many restaurants close after the last lunch reservation (3:30ish) and do not reopen until 5:00 or 5:30.
We are late risers and usually eat dinner late anyway -- 8 PM. We have found that dining when the restaurant is emptying out can be very relaxing. The staff is no longer frantic, the characters come around extra as the number of occupied tables drops, and everyone is more happy to chat. On our January trip to 1900 we had a blast with the characters because there were so few guests left!
JMHO, but those early dinners are often packed with lots of kids, many of whom are cranky from a long day in the parks.