With shutter speed differences, do you mean frames per second burst speed? Actual shutter speed shouldn't be an issue... As it's pretty rare you are likely to shoot at 1/2000th of a second.
Burst rate -- frames per second -- will be an issue if you are shooting sports and such.
Shutter lag may also be an issue, and really comes down to focus speeds. Cameras don't shoot until focus is locked. So a slow autofocus system, or hunting for focus, can create annoying lag between shots. Typically, dSLRs and SLTs will have the fastest focus systems. Most mirrorless cameras and advanced compacts use Contrast detection autofocus, which can be a bit slower. But with a good enough processor and enough focus points, it can be almost as fast as a dSLR. There are a handful of non-dSLRs that use dSLR-type autofocus (phase detection). This is where the Nikon 1 series shines, I think the newest Nex cameras shine here, but I haven't tried them first hand.
Ease of changing settings is huge, if you want more than a "point and shoot." Usually, this is where dSLRs shine. The experience can vary widely between mirrorless and advanced compact systems. Of the models I have hands on experience with, the RX100 does shine here. It has every manual setting you would find in a dSLR, and almost as accessible. It has a handy control ring. It has customizable buttons, so you can set the manual settings you use most. It has a "memory" function for 3 different set ups. Really, the RX100 is more customizable than my dSLR, which says a lot.
Sensory size -- Certainly matters. But careful getting hung up on small differences. For example, my experience with the 1-inch sensor found in the J1 and RX100 -- Much bigger than most P&S cameras, but smaller than dSLR, and smaller than most mirrorless -- It is "big enough" for most purposes. The RX100 handles low light exceptionally well In good day light, IQ is very high, pictures are sharp. Where it doesn't perform like a bigger sensor is in depth of field. It's not easy to get great background blur in portraits, etc. (You can get some, but you may need to get awkwardly close to your subject).
So if taking portraits with nice background blurring is a priority, then you may want a bigger sensor. If that isn't a big issue, then a smaller sensor may be okay with you.
If you are leaning towards putting a lot of weight into being able to change lenses, then the next question is do you already own multiple Canon lenses that you value?
If so, and you don't want to invest in a whole new set of lenses,then you may simply want to look at a newer Canon dSLR like the t3i -- It is a fair amount smaller and lighter than the 40D. Not "small".. but at least a bit smaller. And it would upgrade other aspects too.
If you aren't already tied to Canon lenses, than of course you have more freedom. For example, the Sony A33/35/27 is half the weight of the Canon 40D, and slightly smaller dimensions. (And if you know what to look for, you can get some great old Minolta lenses for Sonys)
I think the first question really is, how small do you want to go?
Personally, I feel I "need" both. I love having something super high quality that can fit in my pocket. Thus, I can get near-dSLR quality anywhere I go. I can take just the RX100 on vacation, and not really feel like I'm sacrificing quality.
But when I do want to be more serious, when I do want to play with different lenses, when I want great portraits, I can pull out my dSLR.
Definitely an investment to have both, but for me, it's worth it. Having 1 "in between" camera wouldn't make me happy. It would still be too big to carry everywhere.
