3 ms·
Why in the world would you design such a system that deliberately fails to explain its decisions?
by tedunangst 11y ago
Why in the world would you design such a system that deliberately fails to explain its decisions?
- pjc50 11y agoClassification. If the system is using all available classified information, then an explanation of its output would have to include the security classification of every input, and the system of classification may also be such that no one person is cleared to know everything. Therefore nobody can know the rationale. (I've made this up, but I think it's the most plausible way such a thing would get built, and it's the logical outcome of "no read up/no write down" security rules.)
- TeMPOraL 11y agoI don't know much about the exact structure of classification. The situation you described would be certainly possible if the classification system is not a tree but a general graph. Is that the case with clearance levels?
- Retric 11y agoYes, it's designed that way for counter espionage reasons. In theory only the president gets full clearance for everything.
- TeMPOraL 11y agoIn the show, Samaritan is a full-fledged AI with its own goals and agenda. In terms of the real world, how many systems do explain step-by-step how they arrived to their conclusions? With some of the analysis and reasoning algorithms, it would be unfeasible to require humans to double-check every step. How would you quickly tell if a Bayesian network aggregating probabilities from hundreds of observations is correct? We don't even require explaining themselves from humans. Instead, we have the concept of trust, which is something one earns. As a commander, I will trust that my subordinate is doing his analysis in good faith, but because there is abuse, we try to have an audit trail that can be used at later date to detect who acted wrongly, and should be deemed untrustworthy. Especially in combat situations, there is no time for an individual to evaluate every piece of evidence. So the point is - are you willing to trust the algorithms you don't understand? With humans you at least share the common mind architecture. In the show, Machine was designed to avoid this question altogether by refusing to make decisions that carry moral weight, only to point out potential persons of interest for human agents to investigate.
- RobertoG 11y agoIt's a plot device. The characters spend their time trying to understand what is going on. But in the context of the script: the designer wanted it to be a 'black box' so nobody controlling the machine used it for it's owns goals. The machine only outputs the minimum relevant information. It's not a bad show.
- thesz 11y agoThere is well known problem with neural networks - they cannot explain reasoning behind their decision. In fact, there isn't any reasoing there.