Smiley has a point, when (if?) they retire the It's a Small World pin it will be worth more. My opinion is that neither IaSW or Casey Jr. will ever be worth the Mad Tea Party pin. The MTP was sort of a surprise item and its retirement came as shock. It ran for the least (so far) of the three Sword in the Stone pins, plus I think the odds were tougher then. The IaSW pin odds have been fairly generous by comparison. I've seen many people with lanyards full of them.
Right now I would think the Mickey Sorcerer hat even by itself is too much for the IaSW pin. Despite a minor re-release as a random prize from Treasure Detection magic during the eBay Dreams promotion last year, those blue ears have been retired close to two years...
The Mad Hatter came out this year and was available for plenty of time during which there wasn't much else to buy so people were flush with credits. It has been retired for several months.
Considering the IaSW pin is likely to retire it should go up a little in value, but it's difficult to say what the pin's post-retirement value will be.
I don't think the pin is worth both hats. I would trade the Mad Hatter by itself. That's fair. After the pin retires, maybe the Sorcerer by itself, but I doubt I'd do that.
As with any trade it also depends on your priorities. If you got extra hats and/or really want the pin, then do the trade. If not, I'd find another deal.
Just my two cents.