I'd agree. I've been there both right after Latvia became independent (beginning of the 1990s) and a couple of times since then. It's changed considerably, especially after Latvia became a member of the European Union in 2005.
First off, for a Baltic city it's rather big (twice the population of Iceland ...). I assume your husband's conference will be in or near the old town, where there are a number of fairly new (or just renovated) international hotels - some of which do indeed have a spa. It's a very nice area, and there's really no comparison with what it was, say, fifteen years ago: a bit of paint and mortar, as well as a number of very new skyscrapers, does wonders for what was once a very run-down and drab area! Today, the old town is quite attractive, with parks and lots of malls and interesting stores selling both name-brand stuff (at more expensive prices then in Western Europe or in the U.S.) or various handicrafts (which are quite cheap, considering the quality).
Since you are thinking of being there only a few days, you should be able to spend all of your time just walking around the old town, visiting some of the churches or the historical places. Beyond the old town, the city is still rather drab. Many of the suburbs, for example, are still very much the concrete wastelands that they were during the Soviet Union area.
The down-side of Riga is that (like many other cities around the world), you have to be careful. Riga has one of the highest crime rates in Europe, and almost one-fifth of the population live in poverty. You and your husband will probably be warned about two things in particular, the scams in the nightclubs (double-check the price of each and everything you order, especially alcohol!), and the pickpockets.
Freyja, you definitely should go. The transformation in the Baltics is really fascinating, and the area has a lot of interesting history!