YES, go!! It is a great party, costume or not. As someone else suggested, you could wear a Halloween themed T, or you don't have to wear a costume at all to have fun.
One year, DD went as Elizabeth Swann from PotC. DH and I didn't have costumes- he just went as-is. I just wore a regular T and shorts, but added a bandana around my neck, a single big hoop earring, and a drawn-on anchor "tattoo" and went as a pirate. I didn't need to pack anything extra, just used things I had already to put together an easy "costume".