Yup. Earlier on in the pandemic, one of the arguments I heard A LOT for why we should't lock down was that our immune systems wouldn't work properly after being confined because they weren't being "used" and that our immune responses for ALL things would be poor after a period of lockdown because the immune system needs to be consistently "activated" and exposed to colds and flu and such in order to work properly. That's what I thought you meant.
The thing about colds, though, there are tons of them and they are constantly mutating and changing (as we've started to see with the variants of our current coronavirus, cousin to the cold) so odd are pretty good you really hadn't ever had that particular iteration of the virus that made you sick.
I have heard of a sort of nasty thing going around. I had it a few weeks ago. Sore throat, cough, the works. I ended up with a secondary infection and needed antibiotics. My whole family had it. I would have thought it Covid, but it started in my daughter a couple of weeks after she had Covid (so the odds were pretty slim she was infected again after just a couple of weeks) and then it made it's way through the whole family, all of us testing negative multiple times. I had one friend with it and landed in the hospital, but she also tested negative on a rapid antigen and PCR. Docs there told her it's going around and they are seeing it regularly. Just a really nasty virus. She also ended up with a secondary infection. My daughter's pediatrician said it's turning into walking pneumonia for a lot of folks.