I've also been patted down when just wearing jeans and a shirt. They weren't baggy jeans but fairly close-fitting ones. I don't know how they thought I could possibly be hiding anything underneath them without it being obviously seen but they completely patted me down and ran their hand along inside the waistband (practically had to hold my stomach in so it could be done since there wasn't a lot of room). So there's also a bit of randomness to it. Just because you aren't wearing a lot of layers you may also get fully searched. However, I would say that the OPs layers probably did raise a red flag.
A number of years ago I got patted down because I was wearing a hoodie. They told me any "bulky" clothes = pat down.
I've posted on here before about being patted down. I'm always careful to dress cautiously when I fly to avoid being chosen for a pat down (no loose clothing, nothing metal, nothing in my hair, all pockets completely empty, etc). However about a year or so ago I still got picked to be patted down. I was wearing thin shorts and a fitted tank top and was barefoot, because I had placed my flip flops in the bin. I still got picked for a pat down.

What's funny is that I assumed they would be checking me from the waist up. Right? Nope! All she patted was my bare legs from the ankle up to the bottom edge of my shorts.


What they thought I was hiding on bare legs I don't know.
I also got patted down several times a couple weeks ago. It was my fault though. We were flying out of DFW Dec 6th and got caught by that big ice/snow storm that hit Dallas. (We were supposed to leave DFW at 6:00am Friday morning and get into MCO at 9:30am. We didn't get out of DFW until late Saturday night Dec 7th and to even do that, they had to fly us to Tampa, where we had to rent a car, and finally got to Bonnet Creek about 2:00am Sunday morning.) To add to our problems, stupid me, was distracted by the weather when I was getting dressed Friday morning. I'm usually really careful not to wear anything that would set the scanners, but wanted to wear my Uggs and without thinking put on a pair of skinny jeans that had leather trim and black studs trim on the back pockets. It was bad enough when I set the scanner off the first time, but dang it, due to the bad weather the tram that takes you from one terminal to another was not able to run all day Friday. AA kept changing our flights trying to get us out of there, but it seemed like as soon as they'd get us on a flight, that flight would be canceled and they'd put us on another always at another terminal. With the tram not running, our choices were to walk around from terminal to terminal or to use their shuttle van. Because we had my 80 y/o mom with us, we chose to use the shuttle van, which meant we had to go back outside security each time, then go back through security at the new terminals. I set off those dang scanners every time I went through them.

I got so tired of those female TSA agents patting my butt down. By the third time it happened, I just turned around and pointed to my butt. The TSA agents would laugh and say. "Oh! butt bling".

We were already having a bad day due to all the flight cancellations and me having to get my butt patted down multiple times Friday just seemed to add to our frustrations. Thank goodness they had the tram running Saturday. In all they had us booked on thirteen flights those two days, before we finally got out.