I've done early entry a number of times during low crowd times and REALLY enjoyed it. Of course, I can't stand crowds so I will take any advantage that I can get.
My travel buddy can't tolerate Despicable Me so I usually send her ahead to do Gringotts a couple of times (a ride she loves and one that I just like) and I do DM and then wander, taking photos. I've been able to get some really cool photos around London and Diagon Alley with no people in them and love to use that time to explore the shops. So far on all my trips I've only ever done the Universal side for early entry although I wish that I could get similar time in IOA/Hogsmeade, an area that gets even more crowded during the day than Diagon Alley.