How on earth can „proprietary software“ with „AI“ in it be admitted as evidence in court ? Are there any third party studies conducted, to test the error rate

How on earth can „proprietary software“ with „AI“ in it be admitted as evidence in court ? Are there any third party studies conducted, to test the error rate of the system ? What’s the algorithm actually doing and what gets annotated by humans? „Move fast and break things“ shouldn’t be the motto when jailing people. This might be pseudoscience payed with tax money!