I would be going alone.
"I'm just very confused. Every time I try to talk to him about it, or I get upset and tell him that I don't really understand why we're having to do this, he defends his family, and says that I'm being very selfish."
Is this the very first time he's done something like this? If so, I'd still be going alone, but maybe he gets a pass.
If it's a pattern, though...if he won't explain things, gets defensive, confuses you, leaves you confused and doesn't seem to mind that...I would be wondering what is going on.
DH and I, before we were married, had some problems. And I could tell when they were cropping up, because he wouldn't make a bit of sense for days and days. Later, I would find out that he was hiding something from me, something he didn't want to tell me b/c he was 'afraid', etc etc. His stories wouldn't line up, HE would say he was confused, etc. DH has done some HARD work in the last 7-8 years (7th wedding anniv tomorrow, but the problems really came to light more than a year before that and in between that time and the wedding he did some serious soul-searching and counseling), but when we were dating and engaged (the first time) he was a fairly big liar, and it was NEVER for a "good" reason. Usually just about money (not usually serious money stuff) and not wanting to be in trouble. He would obfuscate, allow me to be confused, and double-talk (though he was bad at it, so it was more like 1.5 talk ha ha).
Anyway, if this is a pattern with him, I'd be really really careful.
No matter what, though, I'd be going to WDW so that this money wasn't absolutely thrown down the drain. HE chose to take money from savings; I don't care what the reason is, you don't just take money without talking about it with your spouse...he made that decision, and you can make a decision too.