Sounds like algowashing, a.k.a. plausible deniability for the results of their models. Because it's a statistical model and not lines of code, they could go to

Sounds like algowashing, a.k.a. plausible deniability for the results of their models. Because it's a statistical model and not lines of code, they could go to the public and say "This model is sO cOmPlIcAtEd that not even we could have foreseen these results!" when their thumbs are on scales all over the place. I fear what happens when AI criminal profiling models flag undesirables as definitely psychotic serial killers ("just sprinkle some crack on him/some CP on his computer" with extra computer-mediated steps) or when, say, a hospital's AI cancer diagnostic model flags a sponsor's rival as "terminal - hook him up to the MAID machine and run him through the checkout line", and no one can point to evidence that shows the models are being manipulated despite that everyone knows they are.