I've been using LR4 for a while. Here are the key differences for me so far:
1) Only LR4 supports my newest camera. Actually, only the release candidate for LR4.1 does. That alone makes it an obvious decision for me.
2) The image processing in LR4 is better. They made the tone controls work more logically. They also made them work better. In the LR4 class that I taught, people picked up on how to use the controls much more quickly than they did with LR3, so it is definitely more intuitive. I can't honestly say that I've noticed a huge improvement in quality, but I haven't done A/B testing either.
3) The map module is nice. It is helpful if you geotag and it makes it easy to geotag if you haven't been but would like to.
4) I haven't looked at the book module.
In my opinion, it is a nice upgrade and well worth it to me. I don't think that it is a "must have" upgrade, unless, of course, you get a new camera model that isn't supported by LR3 and don't want to go through the hassle of DNG conversion.