LLMs aren't ingesting racial crime statistics, they're ingesting language. The biases LLMs pick up are based on how often a thing is said, not how often a thing

LLMs aren't ingesting racial crime statistics, they're ingesting language. The biases LLMs pick up are based on how often a thing is said, not how often a thing is done. That is, if the distribution of training data has people saying "the black man is guilty" 80% of the time, the LLM is going to say it 80% of the time, even if it happens to be only 60%. Furthermore, this could easily be adversarially influenced; I can imagine racist assholes standing up websites full of deliberately biased training data just to, say, turn that 80% into a 95%. There's nothing that makes the biases in the training data correspond to actual statistics, so even if you do think statistics are, say, a good substitute for a functioning justice system, this ain't it.