I remember way back in the day when they used to clear the parks first. That was back before they reconfigured the entrance to include the fences and bag check areas, and we would be up at the turnstiles with just a sea of people behind us as far as the eye could see, waiting to get in. I honestly don't know how the people in the park got out! They were backed up the monorail ramps and the ferry boat ramp and all the way out toward the buses as far as we could see. We would always arrive early enough that we would be the first or second family behind the turnstiles - we never ended up in the middle of the group.
If I remember correctly, once they couldn't hold that many people outside the gate, they tried letting them in but isolating them in one land until the rest of the park cleared. Obviously, Disney didn't feel this worked well, so they went with the procedure they have now. How they arrived at 4pm for the admission time, I don't know. I skipped a few years of going to the party so I don't know if they tried different times and settled on 4pm, or if it was 4pm from the start of the new procedure.
I can understand some of the frustration for the day guests, but it's not like they don't know that there's a party that night, and when the party guests are allowed to enter. I would plan all my "must do" activities before 4pm, and then perhaps do what another poster suggested and schedule your dinner during the most crowded part of the overlap. Or you could do what I did - buy a party ticket and spend all day in the park, including the party!