I don't see it as ruining the magic. Yes, now I have full familiarity with the Haunted Mansion and know exactly how all the effects work, but I make magic whenever I tell people trivia or things to watch out for. I make magic whenever I work parade control and play around with the little kids.
I once had about 10 of them take me on in a game of soccer during the easter parade. I talked with a family and the sons wanted me to sign their autograph book. I heard about a little girl with special needs and was able to have the Christmas Shop make a certificate with her name on it and candy to bring back. I gave a front of the line pass to a frazzled mother that just arrived in the park late in the day and was afraid she wasn't going to get to go on everything.
Then, when I transfered to characters, it did sort of take away the mystery, but once I started performing it was wonderful.
And like figments said, there are days you don't want to work and fellow CMs you don't like working with, or guests that really make you mad. But then I think about how I was among the few to be selected to learn about attractions and characters and all the memories I've made.