6 ms·

Say in a trial an expert witness is called -- based on the data, was this fire intentionally set or an accident? That expert might be a poor one, but the other

Say in a trial an expert witness is called -- based on the data, was this fire intentionally set or an accident? That expert might be a poor one, but the other side has the chance to cross examine and expose that expert as being unreliable. In the case of this software, though, there is no cross examination. Say the secret sauce is simply a data set and Bayesian reasoning. How do we know if the data set is representative? If it is just an opaque box, that is unacceptable.

Bartweiss 10y ago
The arson example is an interesting one. As far as I can tell, pretty much all of arson science was fundamentally broken for at least 30 years. What were the basic principles of the field - that arson fires burned hotter than others, that puddle marks indicated accelerants, that multiple-origin burns indicate arson - are all wrong. Multiple people were executed for arson homicides on this evidence. Some were executed or even convicted after these principles were rejected. Nor was this a "science marches on" thing. The assumptions had simply been made based on observation and intuition, with no formal testing ever. So this is the sort of thing I mean. It's not a poor expert who can be cross-examined. It's an entire field producing entirely broken results for decades using unquestioned best practices. You can't cross-examine "are even textbooks in your field a complete pack of lies?" and win. I understand the concern, I'm not comfortable with this, but I'm also not convinced that black box software is actually much different from what we have now. The grounds for conviction and sentencing are often completely opaque to the defense already. http://www.newyorker.com/magazine/2009/09/07/trial-by-fire http://www.newyorker.com/magazine/2009/09/07/trial-by-fire https://books.google.com/books?id=GSJ7Ja95oegC&pg=PA382&lpg=PA382 https://books.google.com/books?id=GSJ7Ja95oegC&pg=PA382&lpg=...

Bartweiss 10y ago

The arson example is an interesting one. As far as I can tell, pretty much all of arson science was fundamentally broken for at least 30 years. What were the basic principles of the field - that arson fires burned hotter than others, that puddle marks indicated accelerants, that multiple-origin burns indicate arson - are all wrong. Multiple people were executed for arson homicides on this evidence. Some were executed or even convicted after these principles were rejected. Nor was this a "science marches on" thing. The assumptions had simply been made based on observation and intuition, with no formal testing ever. So this is the sort of thing I mean. It's not a poor expert who can be cross-examined. It's an entire field producing entirely broken results for decades using unquestioned best practices. You can't cross-examine "are even textbooks in your field a complete pack of lies?" and win. I understand the concern, I'm not comfortable with this, but I'm also not convinced that black box software is actually much different from what we have now. The grounds for conviction and sentencing are often completely opaque to the defense already. http://www.newyorker.com/magazine/2009/09/07/trial-by-fire http://www.newyorker.com/magazine/2009/09/07/trial-by-fire https://books.google.com/books?id=GSJ7Ja95oegC&pg=PA382&lpg=PA382 https://books.google.com/books?id=GSJ7Ja95oegC&pg=PA382&lpg=...