I don't know all the intricate details of how these things work, but my general impression is that the terminal should really not be able to see or modify data

I don't know all the intricate details of how these things work, but my general impression is that the terminal should really not be able to see or modify data being sent over pipes. The shell can, but you wouldn't want it unconditionally scraping and "cleaning" what is usually an opaque binary stream (even if it contains mostly text). So you can make it an option, but wouldn't it be better if programs just provided a standard way to turn off color coding?