If you do, aim for later. Last year they kept it open to non-party guests for awhile, and it definitely wasn't less crowded then. In fact, probably the opposite. It was the only part of the park you were allowed to stay in if you didn't have a party ticket, so you either had to leave or go there. I remember it was kind of crazy with a barrier around the area, and TMs everywhere, and huge crowds. They went around checking tickets, and they wouldn't let you stay or enter Diagon unless you had a park to park ticket because they would only let you leave by the train afterwards. I'm pretty sure they gave you a band at this time to signify you were confined to that area and couldn't leave it, but my memory is a little blurry on that. Diagon didn't stay open the length of the party. It only stayed open until what I'm guessing would have been its regular non-party night closing time which was only a couple of hours later for the time of year we were there when the parks close early at like 7:00. It may have been even earlier than that. I can't remember the exact closing time, but we only got a little extra time I do remember that, 1 or 2 hours extra only. Then at that time, they swept Diagon Ally and escorted everyone to the train who hadn't left yet.
For non-party people, it was great to have the extra time in the park, but it definitely wasn't uncrowded. Not sure how it is the rest of the night.