I think it depends on the what area. Criminal Judgement probably shouldn't be a "black box" your going to jail. why? the machine said so. mathematical proofs.

I think it depends on the what area. Criminal Judgement probably shouldn't be a "black box" your going to jail. why? the machine said so. mathematical proofs. My algorithm solved xyz conjecture. it's true. how? I don't know it's a black box. Also, from a diagnosis/debgugging problems it can be a problem. Oh, no the was a huge mistake and the system just catastrophically failed. how do we fix it? don't know. Should be keep using it? don't know. Not saying there aren't places where a black box is perfectly fine.