We have always taken breaks. Probably the biggest factor for us is that we are all introverts.
Being in a crowded/noisy (plus it's usually hot when we go) park all day is really just too much stimulation and we would all be miserable by the late afternoon. We love the parks, but there is something pretty magical about getting onto an empty (air conditioned) bus, breathing a sigh of relief, and riding back to the resort in silence. We probably don't have to go back to the room for 5 hours like we often do, but some kind of escape (riding the monorail, taking a boat or walking to one of the resorts, etc.) is absolutely necessary for our sanity and enjoyment of the parks.
Some families thrive on the stimulation and being around lots of people. If that's you, a break may be a boring waste of time.