On our first two trips with the kids (3 and 8mo, 4 and 20mo) we stayed 8-10 days and took it easy at the parks for the most part and almost always had a mid day break and found we didn't really need to have a "non park"/break day and everyone handled it fine, it was great really. After our last trip though getting used to FP+ and schedules for ADRs and everything, it just didn't work out that we had the midday breaks we were used to. For our next trip we are planning in more downtime and a non park day for this reason, but we're planning a completely different trip in general without a dining plan, lots less pre scheduled, probably most of our FP in the AM, etc. The last trip really wore us out so I do think that having either the breaks or a down day is helpful and I think with enough down time/breaks you'd be fine, but it is a question if you want to enjoy the various things the resort has to offer. For our upcoming trip in addition to midday breaks we're planning on evenings/mornings at the resort in addition to midday breaks rather than rushing from appointment to appointment.