4 ms·
I would like to know how others are dealing with this. I am making a public health system and to deal with this I simply show results to the clinician or social
by arionhardison 4y ago
I would like to know how others are dealing with this. I am making a public health system and to deal with this I simply show results to the clinician or social worker etc... and have them manually approve the result before its applied to an end users protocol.
- NateEag 4y agoYour description sounds like you're training your reviewers to rubberstamp the system's results as correct. That is a horrible outcome. By the time something comes along they really need to question, they'll already have adapted to the system being pretty good and they won't even notice there was an issue. This problem is endemic to ML systems and cannot be patched away, so far as I can tell. We need fundamentally different approaches to avoid this pit. Please make sure decision-makers understand this unavoidable gotcha their ML systems must necessarily have.
- arionhardison 4y agoNot rubber stamp, they can approve or reject the suggestion.
- NateEag 4y agoI didn't mean you were forcing them to approve it. I meant that, over time, your system will become right more and more often, not by virtue of genuinely understanding the situations, but because it has more training data and comes up with correlations that happen to be correct. Once it's mostly-kinda-reliable, your users will, over time, stop paying attention to it and just start trusting it, even if they don't mean to, because that's how humans work. And then it becomes a rubberstamp, and outcomes start getting generated where no human really thought it through.