Sure, you can take them off - with the same little wrench...so put it some place handy.
The medalions are also sold whereever pins are sold, but the biggest selections are in the major pin locations - they are Pin Traders in Downtown Disney, Uptown Jewelers on Main Street in Magic Kingdom, Fronteir Trading Post in FrontierLand in Magic Kingdom, the Sorcerer's Hat in MGM, Pin Central in Epcot (behind Spaceship Earth), Island Mercantile in Animal Kingdom.
The biggest selection will be Downtown Disney unless it's a park specific pin, then you'll want to see the "big" store in each park.
Enjoy!