5 ms·

This is a pretty frustrating read. Interpretability of automatic decision making processes is a genuine, important issue, but I'm not sure this issue is best ex

This is a pretty frustrating read. Interpretability of automatic decision making processes is a genuine, important issue, but I'm not sure this issue is best explained by a writer who, as far as I can tell, has made no effort to understand basic questions like what an algorithm is. The generalized bafflement of the author is just mystification, a vague sense of spooky machines we can't understand, which doesn't help us understand the specific difficulties in understanding the decision making process that depend on processing huge quantities of data.

evrydayhustling 8y ago
Charitably, I think the writing here underscores how much scarier the spread of automation is for people who have never been involved in writing an algorithm (or better yet a complex system). Even if you're well educated in another field, there's a really giant gap between being told 100 times in popular press how information systems work (or don't), versus being part of trying to make one work. Personally I don't see much difference between how opaque the world of automation is and how opaque law, or government bureaucracy, or media production, or any other human industry is from the point of view of a non-expert. But perhaps for many people, it is easier to trust long chains of human processing than long chains of machinery.
- yters 8y ago
  At least with people there is someone you could possibly ask and get an answer. No such luck with a black box algorithm.
  - candiodari 8y ago
    On the other hand, one can verify that an algorithm is fair. One can verify it in the algorithm itself and one can verify that the inputs are fair. In fact, this is often necessary to do just to make it work well. With humans one can't, and as a result there is blatant unfairness everywhere. Nepotism, racism, corruption, ... these algorithms can, if allowed, make it very hard for those to exist. Humans cannot be trusted to make decisions about other humans, not even with the best of intentions (because they still destroy people against the rules to protect the rules). Needless to say a great many politicians are trying to stop the algorithms. Because they can't explain what they've done is the best they can come up with (which is stupid, the better argument, imho, is that one cannot with certainty predict what the algorithms will do in any given case. So if a neural network was the law, there would be no way to know what the edge of legality is. They might send Maria Theresa to jail, so to speak, for tying her shoelaces crossed or some other stupidly small unique thing) Besides, do a bit of law, the reason humans can explain and algorithms can't be explained is as trivial as it is horrible: humans simply don't take more than 2 or 3 variables into account even for critical decisions. This is stupid, for obvious reasons, but it's also why you can explain. I assure you, for a 3 neuron neural network, just taking 3 factors into account, there is no explainability problem. this is a fact often complained about in outcomes. Certain variables make seemingly malevolent decisions innocent. But those variables can't or aren't taken into account, so ... boom hammer comes down, life destroyed.
  - evrydayhustling 8y ago
    Strongly disagree with this line of argument. Almost all human systems are intentionally opaque about some choices or have unplanned consequences you can't get a clear answer about. And, algorithms can be studied in the same ways that we use to critically evaluate human systems -- and sometimes better. Why does your local paper run stories featuring some businesses but not others? Why do college admissions tend to overrepresent some populations and underrepresent others? Why are women paid less on average for the same titles? Why do prosecutors disproportionately bring charges against minority suspects with equivalent weight of evidence? You can't get answers about those human decision processes just by asking - it requires research, competing theories, and sometimes legal action forcing people to share info. Explainability of decisions isn't about how much tech is used... It's about how open a system's operators are to sharing info, and how much effort everyone is willing to put into interpreting it.
    - perl4ever 8y ago
      "Almost all human systems are intentionally opaque" I question the word "intentionally". I would suggest that human systems evolve to be opaque, because that helps them be resilient and survive. If a human system is truly understandable by anyone, then it can be destroyed by some of its members/adversaries and sooner or later someone will.
      - evrydayhustling 8y ago
        Fair point about evolution - what I meant about intention is that some people involved in a decision could tell you more than they are willing to. I think it's worth distinguishing that from systems feeling opaque because of consequences nobody operating them was aware of.
currymj 8y ago
i also found it very frustrating. it reminded me of that famous article about the strange ritual practices of the Nacirema people. (which, if you don't feel like reading it, is actually just an account of 20th-century American life, deliberately made to sound as weird as possible.) it's part of a book, so hopefully the rest of the book doesn't take the same tone.
- rectangletangle 8y ago
  Reminds me a little bit of Gertrude Stein's Tender Buttons, in the book she uses very unorthodox language to make the mundane seem surreal. It has been described as "verbal cubism."
ttctciyf 8y ago
Yeah, this bit for example: > The algorithm was also constantly changing. The data inputs were flowing into the algorithm in real time, but the actual weights, measures and trade-offs that the algorithm made weren’t static either. Some of the functions that the researcher had woven in used machine learning – techniques where the machine constantly learned and adapted to what the most important patterns, correlations and relationships were. It meant that the algorithm was constantly changing and moving as the world moved around it, and its diet of data changed to reflect that. shows the reporter having only a superficial acquaintance, but wanting to sound authoritative, and engenders considerable distrust on my part. For a much much more credible and vastly more entertaining take on the issue of opaque processes empowered with scary real world decision making abilities, see James Mickens' recent talk[1] on why we should be deeply worried by this, enough to update our traditional computer security model (discussed here[2] also.) 1: https://www.usenix.org/conference/usenixsecurity18/presentation/mickens https://www.usenix.org/conference/usenixsecurity18/presentat... 2: https://news.ycombinator.com/item?id=17785162 https://news.ycombinator.com/item?id=17785162
xg15 8y ago
Maybe, but I think that illustrates an important point: He was no expert in how algorithms operate - but neither are the majority of people who are affected by algorithms. I think the warning the article wants to convey is that the current trends could - eventually - lead to a society in which personal freedom and even laws are undercut by completely intransparent algorithms. Intransparent both because companies are allowed to keep them under wraps (and will absolutely do so) and because even if they were made accessible, only a small group of experts would be able to understand them. Sometimes, not even then as the article shows. I think that is a valid concern. The last point - that understandability is such a low priority in development that even frequently the designers themselves don't understand what exactly is happening - is a different problem, but maybe less important than the first one.
- natmaka 8y ago
  It may be a matter of trust and practice. To overcome "intransparency" one may become an expert or trust expert(s) asserting that they did sufficient analysis of the thing in question (here an algorithm) and are confident. In addition, with time, one sees that the thing does an apparently adequate job and doesn't produce so much patent problems. Such things are at first "feared" and rarely used, then gradually gain adoption/traction (benefiting from positive feedback). No one can fully understand the whole technical and organisational context of jetliners, but most of us use them because apparently competent and objective experts continuously check all this, and because problems appear to be rare. This approach may/will fail in catastrophic ways for various causes, either psy (one loses too much autonomy, and incentive) or systemic (the composition of various things may produce an inadequate system).

evrydayhustling 8y ago

Charitably, I think the writing here underscores how much scarier the spread of automation is for people who have never been involved in writing an algorithm (or better yet a complex system). Even if you're well educated in another field, there's a really giant gap between being told 100 times in popular press how information systems work (or don't), versus being part of trying to make one work. Personally I don't see much difference between how opaque the world of automation is and how opaque law, or government bureaucracy, or media production, or any other human industry is from the point of view of a non-expert. But perhaps for many people, it is easier to trust long chains of human processing than long chains of machinery.

yters 8y ago
At least with people there is someone you could possibly ask and get an answer. No such luck with a black box algorithm.
- candiodari 8y ago
  On the other hand, one can verify that an algorithm is fair. One can verify it in the algorithm itself and one can verify that the inputs are fair. In fact, this is often necessary to do just to make it work well. With humans one can't, and as a result there is blatant unfairness everywhere. Nepotism, racism, corruption, ... these algorithms can, if allowed, make it very hard for those to exist. Humans cannot be trusted to make decisions about other humans, not even with the best of intentions (because they still destroy people against the rules to protect the rules). Needless to say a great many politicians are trying to stop the algorithms. Because they can't explain what they've done is the best they can come up with (which is stupid, the better argument, imho, is that one cannot with certainty predict what the algorithms will do in any given case. So if a neural network was the law, there would be no way to know what the edge of legality is. They might send Maria Theresa to jail, so to speak, for tying her shoelaces crossed or some other stupidly small unique thing) Besides, do a bit of law, the reason humans can explain and algorithms can't be explained is as trivial as it is horrible: humans simply don't take more than 2 or 3 variables into account even for critical decisions. This is stupid, for obvious reasons, but it's also why you can explain. I assure you, for a 3 neuron neural network, just taking 3 factors into account, there is no explainability problem. this is a fact often complained about in outcomes. Certain variables make seemingly malevolent decisions innocent. But those variables can't or aren't taken into account, so ... boom hammer comes down, life destroyed.
- evrydayhustling 8y ago
  Strongly disagree with this line of argument. Almost all human systems are intentionally opaque about some choices or have unplanned consequences you can't get a clear answer about. And, algorithms can be studied in the same ways that we use to critically evaluate human systems -- and sometimes better. Why does your local paper run stories featuring some businesses but not others? Why do college admissions tend to overrepresent some populations and underrepresent others? Why are women paid less on average for the same titles? Why do prosecutors disproportionately bring charges against minority suspects with equivalent weight of evidence? You can't get answers about those human decision processes just by asking - it requires research, competing theories, and sometimes legal action forcing people to share info. Explainability of decisions isn't about how much tech is used... It's about how open a system's operators are to sharing info, and how much effort everyone is willing to put into interpreting it.
  - perl4ever 8y ago
    "Almost all human systems are intentionally opaque" I question the word "intentionally". I would suggest that human systems evolve to be opaque, because that helps them be resilient and survive. If a human system is truly understandable by anyone, then it can be destroyed by some of its members/adversaries and sooner or later someone will.
    - evrydayhustling 8y ago
      Fair point about evolution - what I meant about intention is that some people involved in a decision could tell you more than they are willing to. I think it's worth distinguishing that from systems feeling opaque because of consequences nobody operating them was aware of.

currymj 8y ago

i also found it very frustrating. it reminded me of that famous article about the strange ritual practices of the Nacirema people. (which, if you don't feel like reading it, is actually just an account of 20th-century American life, deliberately made to sound as weird as possible.) it's part of a book, so hopefully the rest of the book doesn't take the same tone.

rectangletangle 8y ago
Reminds me a little bit of Gertrude Stein's Tender Buttons, in the book she uses very unorthodox language to make the mundane seem surreal. It has been described as "verbal cubism."

ttctciyf 8y ago

Yeah, this bit for example: > The algorithm was also constantly changing. The data inputs were flowing into the algorithm in real time, but the actual weights, measures and trade-offs that the algorithm made weren’t static either. Some of the functions that the researcher had woven in used machine learning – techniques where the machine constantly learned and adapted to what the most important patterns, correlations and relationships were. It meant that the algorithm was constantly changing and moving as the world moved around it, and its diet of data changed to reflect that. shows the reporter having only a superficial acquaintance, but wanting to sound authoritative, and engenders considerable distrust on my part. For a much much more credible and vastly more entertaining take on the issue of opaque processes empowered with scary real world decision making abilities, see James Mickens' recent talk[1] on why we should be deeply worried by this, enough to update our traditional computer security model (discussed here[2] also.) 1: https://www.usenix.org/conference/usenixsecurity18/presentation/mickens https://www.usenix.org/conference/usenixsecurity18/presentat... 2: https://news.ycombinator.com/item?id=17785162 https://news.ycombinator.com/item?id=17785162

xg15 8y ago

Maybe, but I think that illustrates an important point: He was no expert in how algorithms operate - but neither are the majority of people who are affected by algorithms. I think the warning the article wants to convey is that the current trends could - eventually - lead to a society in which personal freedom and even laws are undercut by completely intransparent algorithms. Intransparent both because companies are allowed to keep them under wraps (and will absolutely do so) and because even if they were made accessible, only a small group of experts would be able to understand them. Sometimes, not even then as the article shows. I think that is a valid concern. The last point - that understandability is such a low priority in development that even frequently the designers themselves don't understand what exactly is happening - is a different problem, but maybe less important than the first one.

natmaka 8y ago
It may be a matter of trust and practice. To overcome "intransparency" one may become an expert or trust expert(s) asserting that they did sufficient analysis of the thing in question (here an algorithm) and are confident. In addition, with time, one sees that the thing does an apparently adequate job and doesn't produce so much patent problems. Such things are at first "feared" and rarely used, then gradually gain adoption/traction (benefiting from positive feedback). No one can fully understand the whole technical and organisational context of jetliners, but most of us use them because apparently competent and objective experts continuously check all this, and because problems appear to be rare. This approach may/will fail in catastrophic ways for various causes, either psy (one loses too much autonomy, and incentive) or systemic (the composition of various things may produce an inadequate system).