We have one, and I'm not thrilled with the experience. DH is a fireman and I work part-time (mostly at home) so we needed someone who could adapt to our weird schedule. The agencies we looked at all charge for one year (you can get a refund if you quit the program during the year, but it isn't prorated in your favor); we went with Cultural Care.
We are on our third au pair since August 2006. The first was just a nutcase. She fawned over DH (not in a sexual way) and didn't get along well with my 4yo DS. So we transitioned her out, and waited over 6 weeks for a new one. Au pair #2 was so sweet, and really good with the kids (also have a 15mo baby). But her English was very poor (it was quite good when we did the phone interview with her - if it was even her), and she had no desire to work on it. If we were gone, she could not communicate even a simple "emergency" over the phone. We really loved her, and if she had wanted to work on her English, we would have kept her. But she refused. So she moved out, and we waited another 6 weeks for her replacement.
Our third and current au pair is really good with the kids. But she's frustrated because she's not making a ton of money (never mind that she knew her salary before she started, and we don't set the salary, the US government does) and she wants to take vacations, go to a private college in the US next year, etc. She also spends a lot of time locked in her room on the computer with friends back home, which isn't positive or productive. Her driving skills are weak, so if she wants to go somewhere, we need to drive her, and she tends to be kind of demanding about when and where we need to drive her. Sorry, with two small children and a husband who works weird hours, I can't play chauffer all the time. Some of the au pairs are decent drivers (especially from Sweden and Germany) but even they need some driving practice with their host parents before taking the car alone. She says that she doesn't want to be an au pair any more, but I think that we may have talked her out of leaving for now.
I think that it is a great concept, and the flexibility and cost are wonderful, but as someone pointed out to me, you get what you pay for. We're not sure that we'll go this route again; the hassle and stress are almost more than it is worth. And with 3 2yos, I can't imagine that you have room in your life for more of that!!