Absolutely, 100%, hands down New Orleans. From your description, you could be my twin sister, and your husband sounds SO much like my ex (well, except I assume your DH's not insane lol). I lived in the French Quarter for five years, and I now travel full-time.
Don't stay on Bourbon Street due to the noise issues. The Monteleone, which someone mentioned earlier, is a gorgeous old haunted hotel with absolutely no noise problems. Or you prefer a bit smaller with a real old-time New Orleans feel, try the Andrew Jackson. The courtyard alone is spectacular! Both are far enough away from Bourbon Street for a quiet night's sleep, but all the craziness is just a short, romantic stroll away.
Visit Bourbon Street because everybody should at least once, but don't spend your whole trip down there. Lower Decatur Street and Frenchmen Street are where the locals go. Spend time in Jackson Square at night--get to know the tarot readers and street artists. Sip coffee, fine wine or absinthe at Pirate's Alley. Have dinner at the white tablecloth restaurants because you can (Jazz Brunch at Commander's Palace is HIGHLY recommended--ask for a table with a cemetery view), but also check out the dives like Coop's (on the aforementioned Lower Decatur). Stroll along the Moonwalk beside the river (but not after 10 pm, stupid legislation!). Have a powdered sugar fight at Cafe du Monde. Take a ghost tour with Haunted History (there are a ton of ghost tour outfits, but they're truly the best). Tour the cemeteries and the River Road plantations. Stay out all night, and wonder at the emptiness of Bourbon Street at 5 am.
As I said, I travel full-time, and have done so for nearly six years. I have yet to find a city that is so permeated by energy and an otherworldly feel as New Orleans is. Halloween is FUN, but if you go then, plan to arrive several days early or stay several days later. You want to get a feel of the place as it is, not just as it is when tons of tourists are in town.