even given that hypothetical being true; (it's misleading in it's most charitable interpretation) A model that makes a prediction based on applying data it was

even given that hypothetical being true; (it's misleading in it's most charitable interpretation) A model that makes a prediction based on applying data it wasn't presented with isn't smarter. It's overfit. Is a model smarter if it's more prone to hallucinating? Given if you point enough examples at it eventually it'll guess right? edit: bonus point, even if you refuse to agree, it'd be an overfit example. A smarter AI would understand the societal implications to both individuals, and trust in the legal system as a whole, and refuse to profile, and make assumptions based on racial identity. You might want to claim, you're asking about probabilities, and using historical data is valid. But then you'd have to explain why data points like "the defendant is black, and black people commit more crimes" would be inadmissible in any reasonable court?