My review of stopping for Despicable Me first:
We did early entry and took the time for Despicable Me before heading to Diagon Alley. Totally worth it! We were on the first ride of the day and done fairly quickly. We had the time to watch the video in the "living room" where you get your glasses and also the training video in the "lab room" before the ride. Of the 4 of us, one got very antsy to get going to Harry Potter and was full of dramatic sighs and an eye roll or two. The rest of us enjoyed the entire experience. The videos may add a few minutes to the time it takes to get through, but they add to the whole thing and its a really fun ride. We still had time to enjoy Diagon Alley before Early Entry ended. DA wasn't crowded and overall it saved us time later in the day when the lines at Despicable Me were long, even with the express pass.
So unless you're only going for the Harry Potter sections, it's worth the stop. You walk right past the ride anyway.