It depends.
The first part of it is how early you go back for your break. We always intend to go back pretty early, but there are days that it ends up being after 2 PM before we leave the park. By the time you get to the hotel, get to the room, and relax for awhile if you have a 5:30 ADR you will be hoofing it back to the park by about 4:30.
Last summer we traveled with friends who wanted to eat at 6 PM every night and we found that it was just too early for us. Of course then you have to factor in the show times for things like Fantasmic if you are planning to see it. It seems like the seatings for the dinner package tend to be really early, but sometimes if you call closer to your trip they have later times open. If you are planning to go to the second performance and you don't care about the "reserved" seating then you can go to dinner later.