Just don't participate in the gift exchange. Let everyone know now that you will not be participating this year. Period. No explanations needed. No one can force you to participate. You are an adult. They can either accept it or they can be upset about it but they can't change it. Only you can.
I don't explain my finances to anyone. It isn't their business. And when you offer up explanations, you open the door for arguments.
This is what I was going to say! Let them know that you will not be giving this year (or any longer if you want to just be done with it), no further explanation needed. It is up to them how to proceed from there. Some may still give you gifts, some may not. Thank those who do and move on.
I became fed up with the whole gift-giving madness a few years ago. It has just gotten out of hand. I have a friend who insists on giving a gift to a group of our friends each year. I used to feel that I needed to give her something in exchange, but I don't any longer. I let her know that I wasn't buying gifts for anyone other than immediate family anymore. Every other friend has done the same except her, but that's her choice.
Good luck! I know it's not easy to change "traditions", but if you can't afford it, or just don't want to spend your time or money in that manner any more then, as others have said, you are the only one who can change it.