3 ms·
Your description sounds like you're training your reviewers to rubberstamp the system's results as correct. That is a horrible outcome. By the time something
by NateEag 4y ago
Your description sounds like you're training your reviewers to rubberstamp the system's results as correct.
That is a horrible outcome.
By the time something comes along they really need to question, they'll already have adapted to the system being pretty good and they won't even notice there was an issue.
This problem is endemic to ML systems and cannot be patched away, so far as I can tell. We need fundamentally different approaches to avoid this pit.
Please make sure decision-makers understand this unavoidable gotcha their ML systems must necessarily have.
- arionhardison 4y agoNot rubber stamp, they can approve or reject the suggestion.
- NateEag 4y agoI didn't mean you were forcing them to approve it. I meant that, over time, your system will become right more and more often, not by virtue of genuinely understanding the situations, but because it has more training data and comes up with correlations that happen to be correct. Once it's mostly-kinda-reliable, your users will, over time, stop paying attention to it and just start trusting it, even if they don't mean to, because that's how humans work. And then it becomes a rubberstamp, and outcomes start getting generated where no human really thought it through.