I don't see specific colors, but I do tend to think of things in tints or light and dark. Memories, people, books, and places are this way especially.
For instance, Harry Potter is very dark, like the sky at Twilight. Navy. Jodi Piccoult books tend to be a sky blue. When I think of middle school it is kind of a drab gray- not bright, not dark. My first college has a brightish yellow tint, like the gel they use when they make Westerns.
It's really funny when I see movies based on books. I've made comments before that the cinematography was not as light or dark as I imagined it. (I'm not talking about the plot, but the actual brightness and the gels they use). I stopped saying that because my friends looked at me like I had three heads.