Are you staying onsite and for more than 1 day or are you buying the EP for your afternoon and that is it? Do you have park to park access? That will make a difference in what you should plan to do.
Personally, I like the atmosphere in the HP areas at night. We did notice that Diagon Alley started to empty out about 30 -60 minutes before close but that was in November so crowds were not that bad to begin with. We are also arriving mid day w/ EP and plan to spend the beginning of our time at US using the EP to ride our favorites then dinner in that park. After that we will wander over to DA I think and see what the crowds look like. We have been through the Gringotts queue before so if the wait is longer than 15-20 minutes, we'll use the single rider line. That was perfect on our last trip, never waited more than 5 minutes. You do at some point want to experience the bank lobby and elevator but you can always save that for the next morning with early entry if you are staying for a few days. Or, if you have PTP access, and DA is feeling too crowded, you can take the Hogwarts Express to IOA and tour Hogsmeade and see the castle at night then work your way out of IOA using your EP.