For us, it was less about the meal times themselves, and more about what else you plan to do each night.
Do you plan to go to the main shows each evening? If so, then if you allow about 15 minutes before the show starts to get there and get seated, it is really only about 15 minutes extra time than if you were going to the MDR for main dining.
Also, there are a bunch of activities between the two dinings, we found it was easier to go to these if you had already gone to dinner, otherwise you kinda had to cut them close to go them and then get to late seating dinner on time.
Likewise there are some activities scheduled for after second seating which can be a little rushed if you are trying to get out of second seating.
On the other hand, if you don't go to a lot of the shows/activities, late dining probably is less rushed.