So it's been a long time but I figured I'd give an update -
Caveats: I'm an amateur. nuff said.
I swapped out my Canon 18-135 for the 24-105L f/4and also added a 70-200L f/4IS.
I absolutely love both lenses.
First off if you are considering a 70-200mm imo you MUST get the IS. I am getting vey sharp photos at 180mm+ and shutter speeds at 1/80, 1/100 etc and that just isn't possible off tripod without IS. I tried to save a few bucks by getting the non-IS and I promptly returned it and got the IS. Don't waste time, just get the IS.
That 70-200mm is amazing but that's no secret I guess. All 4 of the 70-200mms from Canon are super highly regarded and so that's that. If you need/like this focal length just do it and you'll be thrilled.
Both lenses are sharper than any other non-L zoom Canon makes...even the 17-55 2.8 IMO (non-L primes may very well compete though). You can see this, objectively, on the test site linked above. Whether or not you will see this in your own photos is subjective of course but I have for sure. to me, the 24-105 is a much better lens than the 18-135 in both sharpness and color.
when I was thinking about all this I tended to downplay "build quality". I just figured I wouldn't care all that much. Well I can say now that these lenses make other lenses feel like plastic toys. They are solid and substantial, they look nice (red ring, LOL) and they focus silently and quickly with nary a zzzzt or whir to be heard. Very satisfying. The 70-200mm focuses internally and doesn't change length so even nicer.
I will say if you are in the 18-23mm range a lot, you will miss it, but that's obvious.
As for being at f/4 max...well first off it's nice to have on the zoom end for sure. I personally have really started to use moderate ISO/bounce flash indoors with nice results so I haven't missed the f speed (work in progress though, sometimes my lighting is uneven but to me it's much better than higher ISO no flash or, *gasp*, direct flash). I'm starting to get addicted to photos with that flash twinkle where a casual viewer does not even know a flash was used.
I also enjoy the dof...when I was trying to avoid flash I was using my nifty fifty or my since sold 17-55 2.8 and opening it all the way up and it was hard to get everyone around a table in sharp focus. Plus even wide I was at 1/60 or 1/30 and again it's harder to be sharp. Stop down, crank up to ISO 400/800, go 1/100 or higher, bounce a flash and I'm in happy mode indoors.
So yeah, I'm thrilled.
Is it worth the $$$ to buy these lenses? Well that's a super duper subjective question. It was to me.