The use of "colored" to describe the hue of an object predates its use to describe certain skin pigmentations by several hundred years.[1] After the second use

The use of "colored" to describe the hue of an object predates its use to describe certain skin pigmentations by several hundred years.[1] After the second use was invented the former still stayed common, eg in "colored pencils", "colored clothing", "colored contact lenses", etc. If you allow any term which has ever been used in a racist manner to be considered offensive you give enormous power to racists. They get to DoS attack your language by using whatever they want as slurs. I don't like empowering racists. [1] https://www.etymonline.com/word/colored https://www.etymonline.com/word/colored