3 ms·
This reminds me of a talk that I saw about wavelet based algorithms in the 1990s for detecting tumors in mammograms. The algorithms found most of the tumors th
by btilly 10y ago
This reminds me of a talk that I saw about wavelet based algorithms in the 1990s for detecting tumors in mammograms.
The algorithms found most of the tumors that humans had missed, with similar false positive rates. BUT humans refused to work with the software!
The problem was that the software was very, very good at catching tumors in the easy to read areas of the breast, and had lots of false positives in more complicated areas. Humans spent most of their effort on the more complicated areas. Every tumor that the software found that the human didn't simply felt like the human hadn't paid attention - it was obvious once you looked at it. The mistakes felt like stupid typos do to a programmer. But the software constantly screwed up where you needed skill. The result is that humans learned quickly to not trust the software.
- antognini 10y agoThis is very true and directly related to my research. (I work at a company developing software to interpret EEG data.) There's a huge difference between an algorithm with a low error rate that makes mistakes seemingly at random vs. an algorithm with a somewhat higher error rate whose mistakes are at least comprehensible. A doctor is much more likely to trust the latter than the former. Almost as important as developing a detector with a low false positive rate is developing a detector that can figure out when the problem is too hard so it knows not to even try. (And it seems that this problem is just has hard.) One of the things we do is perform a Turing test of sorts where we test if the performance of our detector is statistically indistinguishable from a human. (In fact, we actually have a contest running right now where we give you 10 EEG records, some marked by humans, some marked by our software, and if you can figure out which were marked by which we'll donate $1000 to the American Epilepsy Society.)