7 ms·
The hidden reality of ridiculously complicated algorithms
- jtbayly 8y agoIf I was the guy who risked my job (or more) to get this out there, I’d be ticked that this is all the journalist managed to produce with it.
- tedunangst 8y agoYeah, I'm finding it hard to believe this guy will never work in tech for the rest of his life.
- growt 8y agoBut he shared the terrible secrets that there are "functions" and "libraries" and different colored words.
- AlexCoventry 8y agoThat came across as such cynical sensationalism, too. Unless there's more to it than revealed by this excerpt, it's not like he'd lose his job for revealing a terrible secret, or anything, it would simply be holding him to account for the professional lapse of violating his confidentiality agreement.
- ggm 8y agoAnyone remember back in the 1980s when a UK medical school tried expert systems to do admissions and built in systematic bias? Oh wait.. didn't that just happen again in Japan? And here we have the tls discussing heuristics to sentencing which "out performs" the judge...
- temporallobe 8y agoWe have "expert systems" now in medicine. It's called the insurance EHR. Doctors punch in a diagnosis, system spits back what treatment and/or medication the insurance company is willing to pay for. Even if the doctor completely disagrees, that becomes the de facto solution for the patient.
- culturedsystems 8y agoThis is a pretty frustrating read. Interpretability of automatic decision making processes is a genuine, important issue, but I'm not sure this issue is best explained by a writer who, as far as I can tell, has made no effort to understand basic questions like what an algorithm is. The generalized bafflement of the author is just mystification, a vague sense of spooky machines we can't understand, which doesn't help us understand the specific difficulties in understanding the decision making process that depend on processing huge quantities of data.
- evrydayhustling 8y agoCharitably, I think the writing here underscores how much scarier the spread of automation is for people who have never been involved in writing an algorithm (or better yet a complex system). Even if you're well educated in another field, there's a really giant gap between being told 100 times in popular press how information systems work (or don't), versus being part of trying to make one work. Personally I don't see much difference between how opaque the world of automation is and how opaque law, or government bureaucracy, or media production, or any other human industry is from the point of view of a non-expert. But perhaps for many people, it is easier to trust long chains of human processing than long chains of machinery.
- yters 8y agoAt least with people there is someone you could possibly ask and get an answer. No such luck with a black box algorithm.
- candiodari 8y agoOn the other hand, one can verify that an algorithm is fair. One can verify it in the algorithm itself and one can verify that the inputs are fair. In fact, this is often necessary to do just to make it work well. With humans one can't, and as a result there is blatant unfairness everywhere. Nepotism, racism, corruption, ... these algorithms can, if allowed, make it very hard for those to exist. Humans cannot be trusted to make decisions about other humans, not even with the best of intentions (because they still destroy people against the rules to protect the rules). Needless to say a great many politicians are trying to stop the algorithms. Because they can't explain what they've done is the best they can come up with (which is stupid, the better argument, imho, is that one cannot with certainty predict what the algorithms will do in any given case. So if a neural network was the law, there would be no way to know what the edge of legality is. They might send Maria Theresa to jail, so to speak, for tying her shoelaces crossed or some other stupidly small unique thing) Besides, do a bit of law, the reason humans can explain and algorithms can't be explained is as trivial as it is horrible: humans simply don't take more than 2 or 3 variables into account even for critical decisions. This is stupid, for obvious reasons, but it's also why you can explain. I assure you, for a 3 neuron neural network, just taking 3 factors into account, there is no explainability problem. this is a fact often complained about in outcomes. Certain variables make seemingly malevolent decisions innocent. But those variables can't or aren't taken into account, so ... boom hammer comes down, life destroyed.
- infinity0 8y agoThe algorithms are opaque but how's that different from a human's own brain? We don't have a very good view into a judge's own life experiences from which his own decisions are based either. At least with an algorithm we have a chance of seeing how it really works, in principle. All the data is recorded and the code is available. > The reality is that if the algorithm looks like it’s doing the job that it’s supposed to do, and people aren’t complaining, then there isn’t much incentive to really comb through all those instructions and those layers of abstracted code to work out what is happening.” If a judge is performing well are you going to run an expensive background audit on him, or open up his brain to try to see how he "really works"? The difference is that algorithms today are practically run by centralised organisations that control it plus the data, rather than having lots of humans all slightly different making their own decisions in a semi-centralised[^] distributed fashion. But the opaqueness is nothing new. [^] because most human organisations today are hierarchical, but at least most people have some sense of self-agency.
- A2017U1 8y ago> All the data is recorded and the code is available. PredPol as mentioned in the article is closed source and considered a trade secret not viewable by the public. It's the largest supplier of these tools in the world.
- infinity0 8y agoHuman brains are even more closed, than closed source software. That was my point. With closed-source software, at least someone has access to the code and data.
- coldtea 8y ago>The algorithms are opaque but how's that different from a human's own brain? Algorithms can blindly follow through into an obviously illogical decision in ways that a human brain doesn't (and when it does, we consider that person unfit for a job as well).
- rs86 8y ago
- malmsteen 8y agoThe terrifying reality of articles with way too much words 8x
- gjstein 8y ago> “We need”, Jure said emphatically, “to step up and come up with the means to evaluate – vet – algorithms in unbiased ways. We need to be able to interpret and explain their decisions. We don’t want an optimal algorithm. We want one simple enough that an expert can look at it and say nothing crazy is happening here. As someone who works with machine learning, I find this statement rather misleading. The researcher quoted here has a very strong bias about what it means for an algorithm to be "optimal". A system cannot both "be optimal" and also "biased" in a way that the designers don't like: such a system is indeed not optimal. The dialog in the machine learning community has increasingly been about how we might structure these systems in a way that they are unbiased; it's a shame the researcher (and the author of this article) seem to think that "simplicity" is the only option. (In case anyone is interested, I recent wrote a blog post to this effect: http://www.cachestocaches.com/2018/7/bias-and-ai/ http://www.cachestocaches.com/2018/7/bias-and-ai/ )
- slv77 8y agoEvery model designer is going to come to the table with his own set of lenses and biases based on this own life experiences. These biases are incorporated into our models based on the metrics that we optimize for and the features that we incorporate into our models. It may comforting to think that all our biases are “baked out” of the models during the training process but if the researcher had been a black man do you believe that they would have arrived at identical models? Even if the model designer found ways to compensate for his own bias the world that these models operate in are inherently biased. Training data may be subtly biased in ways that are difficult to detect. A model trained on recidivism is likely to be biased if, for example, a lower class black male or upper class white women have different risks of being arrested for the same crime. Assuming the models themselves are unbiased as trained they may create bias with subtle errors when pushed into production or due to second order effects, feedback loops or errors or manipulation of data. For example during the credit bubble entire industries found ways to raise borrowes FICO scores. There is also the inherent corporate bias that the researcher pointed out which the bias of the almighty dollar. Compensating for bias is expensive and may even impact salability if an unbiased model doesn’t “gel” with the expectation of customers who are biased. For example a judge may feel pressure of homogenize his decisions or abdicate his responsibility by rubber stamping the models decisions. Currently we compensate for this in a democratic society with checks and balances. What are the checks and balances for a computer algorithm that is a trade secret for a for-profit corporate entity?
- sabujp 8y agohttps://cs.stanford.edu/people/jure/ https://cs.stanford.edu/people/jure/
- rahimnathwani 8y ago"And there it was: a white screen with instructions neatly arranged in a series of boxes." If seeing a Jupyter notebook impresses the writer so much, it might blow their mind if they found out that the screen is made up of millions of dots. And that each dot is actually made up of three different-coloured dots.
- jzwinck 8y agoI too realized the author was looking at Jupyter when I saw In[3] and so on. Absurd. The author is conflating a typewriter with a story. And starting from such ignorance, I wasn't about to find out where it ended up.
- deleted 8y ago[deleted]
- jimnotgym 8y agoI have to challenge the statements of Jure Leskovik > They often have no feedback on whether they made the right decision Of course they do. They will see people stood in front of them that have violated their parole all of the time. They can read their case notes and use the information to adjust future decisions. This of course is after a career of standing the other side of the bar as an advocate/attorney of those people, because that is what you have to do to become a judge in the first place. > and there is no knowledge-sharing between the judges. Except the whole legal system in the US is built on common-law. Literally the law being applied in a common way to everyone. Literally by knowledge sharing. If the esteemed academic is suggesting that parole decisions are being shared less well than judgements the fix seems to be rather simpler than ML. The idea that a ML algorithm is somehow less biased than a judge is of course just as absurd. A ML researcher is just as capable of using race, for instance, as a characteristic. They are just as capable of seeing a correlation without understanding the complex underlying causes. I choose the human judge.
- andrewaylett 8y agoOne of the interesting -- or at least, I hope it'll prove to be interesting -- effects of GDPR is that we have the right to an explanation of important automated decisions and also the right to human review. Before anyone comments that ML models are notoriously difficult to explain, the whole point of the regulation is that opaque models are intolerable. Also, https://github.com/slundberg/shap https://github.com/slundberg/shap looks like it could be very useful.