It's hard to say what day of the week will be less busy as most of the state of FL has the week off for Spring Break.
But as far as getting there first thing in the morning is concerned, YES! The lines (For Hippogriff, Dragon Challenge and Forbidden Journey) are *typically* shorter then. Olivanders will always be long, so that's something everyone has to deal with. But if you get through those rides first thing, you'll have time to do Olivander's and then take in the rest of the sights.
Also, if IOA is the only thing you are doing that day, and planning on being there all day; I recommend going back at the end of the day to see Potter at night. It's very cool, and sometimes the lines get short again right near park closing time.
Have fun!