First the term was neutral. Then the word became a widely used pejorative. Nowadays the word is hardly used at all. Why did people stop using "colored"? Because

First the term was neutral. Then the word became a widely used pejorative. Nowadays the word is hardly used at all. Why did people stop using "colored"? Because it had turned into a pejorative. When did that happen? Decades ago. That's why I said its historical usage is pejorative. This isn't controversial. The word wasn't conceived as a pejorative, but that's utterly irrelevant in this context, so there's no point in even bringing it up. You could also point to a time in history where the N-word wasn't technically a pejorative. It originally (and etymologically) simply meant "black". Would you also argue against the claim that "it's widely known that the historical usage of the N-word is pejorative"? I sure hope not.