No, it is only a slur when it is intended to be. Saying that kids who climb around are little mokeys isn't the same thing.
If it is meant to be offensive, as is obviously the case with the sports guy, our outrage and condemnation ought to justly fall on the speaker.
If it is said about goofy little kids, with no racist intent, then there is nothing wrong with it. If someone gets offended because they can't understand the difference, that is their problem. I'd just leave them to be offended.
The phrase, "people of color" is nonsensical. Of course everyone is some color. What people mean when they say, "people of color," is people who are black, latino, or (sometimes) middle eastern. They don't generally count any other caucasian people or anyone at all of mongoloid descent.
Personally, I'd like to see phrases like "people of color" disappear. They do nothing but divide people based on skin color.