We have animal and other allergies and asthma in our family, and I never wash clothes away from home, but that is because 2 of our childrens' issues are a constant battle to keep under control. I don't know what your son's allergies are or how severe, but I know washing alone doesn't eliminate the allergenic properties in animal danders or if he's allergic to, say, peanut oil residue. When we travel, I just pack enough clothes, but I bring a small travel tube of our detergent in case I have to wash something in the sink.
ETA: We always get an allergy cleaning so they wash all the bedding in hot water separately with, I believe, unscented products. Sometimes my one DS still ends up a little itchy, so I have to give him benadryl before bed (which I don't like to do) then shower when he wakes up.