I have not had a problem with my pin bag, but on my return flight from WDW, they did search my backpack because there was concentrated metal.......basically the little bags and bags of pins I bought!!!!!!
The only thing I will do different is to remove my pinbag and my bags of pins and put them on the belt that way......then I can repack....it was kind of icky to have him going through my backpack (when I go to WDW that is essentially my purse as well) since the way it was packed, there were personal things in there as well! Plus I kind of felt everyone was looking at me!
As for your question about carry on vs checked bags. I have always had my pins on my person. I don't like leaving them out of my sight (they may end on vacation without me!), plus if there is going to be any bag searching going on, I want to be there to watch.