We were at DL the same days you were. I would consider June 4 and 5 to have been pretty low crowds. We found that the lines were fairly short for everything (except Star Tours!) The 6th was significantly more crowded.
I don't have much to add to your question about a crowd comparison with spring break trips as we have not done that, except to add the question of what is considered Spring Break season at DL? When we were there last week, an AP holder was telling me that Spring Break season at DL starts the first of April and that DL is not very crowded at all in March. Is that true? Around this part of the country our spring break happens in March, and I know that WDW is absolutely packed the last week or two of march with the spring break crowds. If that is good information, then maybe we will consider a spring break trip too!