I was tested about a year and a half ago, adult onset allergies, and milk was the only reaction I had in the food tests. It was a slight allergy, nothing to really worry about, the doc said, but I do notice that I seem to have a small reaction when I do drink milk or have some kind of dairy.
I tend to have a slight (very slight!!) tickle in my throat if I'm drinking a whole cup of milk or bowl of ice cream but it's not enough to make me stop.
My doctor wasn't concerned about it either, he told me to just not drink a whole half gallon of milk or tub of ice cream.
On a scale of one to four, which is what my doc uses for reactions, I rated a one.
I hope this helps.