With a line as long as Hagrid's, there could definitely be a scenario in which staying in line even if it goes down is less total time than leaving and coming back later. If I started hearing rumors that it was down I'd stick it out until an actual cast member told me that it was likely to be down for hours.
Meanwhile, the good news is that with Express Passes, you don't have to worry much. Our visit on MLK weekend might be lower traffic than your Easter visit, but we never waited more than 15' or so for any ride except Hagrid's and Velo. I'll be the voice of dissent and say, if you must bail on Hagrid's, don't go straight to Velo, since that's what everybody else will be doing. Velocicoaster wait times tend to be much shorter than Hagrid; 60 minutes is a short wait for Hagrid but a long wait for Velo. Also, Velo times actually go down as the day goes on, while Hagrid usually stays steady. I say let everybody else rush to Velo, but use the app's "notify me when the wait is below" feature, set for something like 50', and ride it later.
Instead, since you're nearby, maybe do Forbidden Journey, unless it seems like everyone is rushing that too.
Having Diagon Alley more or less "to yourself" is very nice. That's what we found when we arrived within two hours of park closing time, so you might try that. You could also start your second day there.
