8 ms·
I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, f
by crazygringo 7y ago
I think it's important to note that human pattern recognition is basically black-box as well.
We can't "explain" how we recognize a particular person's face, for example.
Robust pattern recognition based on 100+'s of factors is just inherently black box.
Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.
Black box doesn't necessarily mean bad. I think the relevant question is: how do we ensure machine learning is trained and verified in ways that don't encode bias, and only used with "intended" inputs, so a model isn't being trusted to make predictions on things it wasn't trained for?
And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?
- fhennig 7y ago> because despite being less accurate, they can have designers who can be held legally accountable and liable Are you implying you cannot be held accountable for black box models?
- darkerside 7y agoI believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?
- danShumway 7y agoI'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain. AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for using the system. You can still hold manufacturers and businesses accountable for promising a degree of accuracy that their system couldn't meet. In the same way, if my dog bites someone, and I get sued and argue, "animal motivation is really hard, we don't know why the dog chose to do that", I'm not going to win the case. The reason it happened is because I didn't leash my dog. The reason the black box hurt someone is because an accountable human improperly used it or inaccurately advertised it.
- xmly 7y agoThere are payment systems based on human facial recognition. If the model failed, it means losing money. Apparently you can not deny liability because of using a black-box model.
- lonelappde 7y agoThis is tort law, the worst defined laws we have. Liability is far more complex that and usually governed by irrational humans or arbitrary bureaucratic rules contrived by powerful people protecting their interests.
- danShumway 7y agoBut how do black boxes change that? It's still the same people operating under the same laws. A few commenters here posit that without a black box, designers might be held accountable. If there's a system where software manufacturers would be held accountable for bugs, is any judge going to say, "well, the software is bugged, but they don't know how to fix the bugs, so they're off the hook"? If there's proof that Twizzlers are poisonous and kill people, the Twizzlers manufacturer can't say, "but we don't know why, we just threw a bunch of chemicals in a vat at random without writing down the labels. So therefore, it's not our fault." I don't know, can they? I'm not a lawyer, maybe everything in our legal system is way more horrifyingly broken than I assume.
- JauntyHatAngle 7y agoAn anecdote, but this is a criticism being aimed at YouTube currently for it's demonitisation scheme. The argument being made is that even though the demonitisation of certain classifications of content is immoral (e.g. demonitising any and all LGBT content, even when family friendly), it is tacitly being allowed to happen due to YouTube's stance of "the algorithm" making the decision, not them. Which may lead then to have a position of "supporting" LGBT as a company while simultaneously being able to demonitise said content, which could hurt their business in actively hostile to LGBT countries. This isn't proven, but it's an example of how it could be used to absolve yourself of blame and point it at an inscrutable system.
- MaupitiBlue 7y ago> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction. Isn’t the first sentence just an example of the second?
- MaxBarraclough 7y agoIt's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions. That's not the same question as whether there were feelings involved, or whether the process was conscious.
- perl4ever 7y ago"The point is that the explanation given does not correspond to the process through which the decision was truly reached." Sure, but that doesn't mean "a weighted combination of feelings" is an accurate description of the true process. That's tendentious and speculative. What I think we can be sure of is that there are parts of our decision making that we are not consciously aware of, but precisely because of that, one should be cautious of assuming how it works even if you're not doing it by reflex. Psychological explanations are the rationalizations that people make after the fact.
- lonelappde 7y agoParent wasn't saying that the weighting process was rational. The weights are also feelings.
- perl4ever 7y agoAs long as we don't know what the weighting process is, we shouldn't make unfounded statements about how it works. You're asserting that it exists, but nobody knows anything about what it is. It's a program that you haven't read the source code for.
- devit 7y agoHumans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.
- sojournerc 7y agoBut that explanation is likely full of cognitive bias, and all sorts of subjective experience. Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.
- Balgair 7y agoObject recognition is firmly in the realm of Neuroscience. We've had a good idea what the eye-V1 path is doing in mammals for a few decades now. To the point where we are implanting chips into the brain to (crudely) re-establish vision in blind patients. Though the research is very much on-going, it is being better understood with each passing year.
- sandpaper26 7y agoRespectfully, unless I'm not understanding what you mean by this, that seems very far from object recognition. You're replicating the data stream that would normally be coming from the optic nerve/retina, but that's not the same thing as understanding how the data of an object's image relates recognition of that object. If I've never seen the entire Mona Lisa before (but maybe have seen portions of it), your proposed system could force me to see the painting but it can't force me to recognize it as something I have seen, nor force me to know that it is a famous painting by Leonardo da Vinci currently on display at the Louvre.
- Balgair 7y agoI don't think you are understanding. But Bio isn't meant to be easy. At each synapse a significant amount of data processing occurs, right from the rod and cone cells in the retina and at nearly each synapse along the way. 'Data' is more-ish conserved into V1 where all hell seems to break loose, path-wise. After V1 and in nearly the entire cortex, these data are distributed in unique and very complicated ways throughout the brain. Most of the mammalian eye-V1 pathway is well understood, but that process is too much for a comment on HN. But, I want to stress, we have a very very good idea of what is going on. That said, the research seems to indicate (highly debated still) that there are specific synapses that encode 'the left side of grandma's nose in dim lighting from 10 feet away' and other many other things. Again, I cannot stress enough that such 'grandma' synapses are not firm science and there is a LOT of research still ongoing. But the evidence seems to be pointing that way at this time. Such cells are fed this information from V1 and, likely, a lot of other places. When such 'grandma' cells fire, they then send that signal out to other cells they may be connected to. Such a system is likely replicated many times all over the brain; there are many copies of 'grandma' cells and they are wired up in many different ways. The physical location of 'grandma' cells in highly unique, if they exist at all. Again, research is very much on-going.
- daoxid 7y ago> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".
- knzhou 7y ago> But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie. > So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?". You can already do that. Just change that number in the input and see how the output changes. To that extent, even the most black box AI model is more transparent than human decision making.
- daoxid 7y ago> You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie. Well, I guess it depends on how self-aware a person is. I think the biggest danger is trying to rationally explain your decision when in fact it was based mostly on your feelings, in which case I agree that the explanation is "basically a lie". One needs to be honest when something is not based on a fact but on a feeling to prevent pointless discussions. (If I hold an opinion based on a feeling then you cannot convince me that I am wrong by giving me facts.) > You can already do that. Just change that number in the input and see how the output changes. Makes sense. But I guess transparent models would still be generally preferable because you can fully understand how the output is produced, whereas in black-box models you might have to ask quite a lot of questions to get a feeling for it, but even then you can't be sure that you have a full understanding of it.
- chasedehan 7y ago> And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations? Decision trees are black box models and are highly accurate. (Think XGBoost, LightGBM, Random Forest, etc) Not sure what point you are trying to make here.
- gusostow 7y agoThose models are ensembles of decision trees. A single tree is interpretable, but sensitive to overfitting.
- Balgair 7y ago> I think it's important to note that human pattern recognition is basically black-box as well. Mammalian pattern recognition is fairly well understood. We have a good understanding of the information path from the eye to V1 in most model animals. So sure are we of this pathway that we've successfully implanted ships into blind patients and restored sight (results vary a LOT, though). I know this may not fit pattern recognition in a super-philosophical sense, but damned if it's not a ways down that road. We're literally shocking a person's brain so that they 'see' again (!!!). How V1 feeds into the rest of the brain is VERY active research and has had a lot of successes, but we're a long ways from a 'black box' these days. Honestly, the auditory pathways are a lot better understood, as they seem to be 'older' and specific tones are represented physically in the mammalian brain (tonotopy). As such we know a lot better (relative to sight, smell, taste, etc) how tones are encoded and how they elicit specific recognitions and responses (this work is NOT straightforward). In Neuroscience/Biochem, we are getting a LOT closer, but yes, we are not there yet by any means. The 'black box' idea of the mammalian brain is not going to last another 500 years, likely. We're marching along with good time as a species.
- username90 7y agoKnowing how to input data to the visual cortex doesn't mean that we know how it works. It is the same with neural networks, we know how to pass data to them and how they do the math, but we can't trace back its reasoning to know why it comes to the conclusions it does.
- Balgair 7y agoWe have a very very good idea of the mammalian eye-V1 pathway. It's not a straightforward path and describing it in a HN comment is not going to be useful to anyone. Explaining the complex, but well known, pathway is a job best done by other people than myself. We are well aware of what 'data' processing the mammalian brain does at nearly every synapse, and it is a LOT. Right form the cones and rods, data is being highly processed (synapses are VERY noisy, as it turns out). So while more research always comes up with more questions, we are fairly confident about the eye-V1 pathway (contrast that to other organs, like the vestibules, where we're fairly in the dark still). That said, wait, what? We can't do the back-propagation calculus in a NN? When has that been true and for how long? I thought it was fairly straightforward to know the weights of the connections of the nodes in your network. It's just a tensor you grep for, right?
- inimino 7y agoSure, human pattern recognition is poorly understood, but it's a system that evolved over millions of years. We should expect more from something we ourselves designed. The question of why we would choose a black box model over something interpretable that works equally or nearly as well still needs to be asked, especially if there is a misconception of black box models working better than interpretable models due to current trends and the hype cycle. > held legally accountable I think this misses the point. It is not about accountability but about getting the right results. The right question is where we need an interpretable model to eliminate biases or otherwise satisfy ethical requirements. Often you can't know whether a decision was ethical without knowing how it was reached. Would you accept a trial verdict given by AI? What about a lending decision, or a job promotion? If you need to justify a decision and not merely reach a decision, you need something interpretable.
- 8bitsrule 7y ago>human pattern recognition is ,,, a system that evolved over millions of years. We should expect more from something we ourselves designed. Why? Human intelligence evolved over millions of years. And we're struggling to keep our changes to the world from killing us.
- inimino 7y agoBecause we design it, so we have the ability to choose its characteristics. The current prioritization of 1% improvements in artificial benchmarks over all else is a disease that is actively retarding progress in the entire field. Political systems are also designed. If human behavior is in danger of killing us, we must look to the political systems we have created and solve the problems there.
- Frost1x 7y agoI think it's also important to remember a lot of human models were abandoned because of the inherent bias and black-boxed nature of them (feelings, subjectivity, etc.). This is one of the reasons the scientific method blossomed: objectivity, rigor, transparency, reproducibility, etc. Black boxes can lead to bad decisions because it's difficult to question the process leading to decisions or highlight flaws in conclusions. When a model is developed with more rigor, it can be openly critiqued. Instead, we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases. To be clear, I'm not saying ANN/DNN models are bad, just that we need to understand what we're getting into when we use them and recognize effects that unknown error bounds may cause. If, when the model fails to correctly classify a new data point, the result is your photo editing tool can't remove a background person properly... then so be it, no harm no foul. If the result is that the algorithm classified a face incorrectly with high certainty and lands someone in prison as a result (we're not there, yet) then we need to understand our method has potential unknown flaws and we should proceed carefully before jumping to conclusions.
- galaxyLogic 7y agoThat is the fascinating thing about consciousness. How do we "know" we see red and not green or vice versa? How do we "know" we are feeling heat or coldness? We just do. We feel different things in different, yet recognizable ways. And these "feelings" are multi-dimensional, feeling heat and feeling color are not on the same axis. Red and Green seems to be on different points of the same axis and hot and cold are points on some different axis, other dimension. How many different dimensions of "feelings" are there? Note they are not the same as different senses. There can be many different feelings associated with things we see and recognize that way: colors, and shapes and lightness vs. darkness etc.
- jotm 7y agoPhysical sensation seems simple tbf. Any robot can sense temperature or color gradient. It's the decision making that's complicated. To make a decision on whether it would like chocolate ice cream or not (before trying it) a brain will take all the data gathered over time, cross reference it with related data (does chocolate taste good alone? \ did I like it? \ It looked rather inedible. \ was it high cocoa content or not? \ Chocolate yoghurt seems similar, but I did not like it. \ I do like cold, but only when it's hot outside. \ My friend says chocolate ice cream tastes good. \ My other friend says it tastes bad. \ trust or discard opinions? and much more). All within minutes, seconds even. An AGI would need an absolutely massive database of knowledge similar to the one acquired by a human over the first 20-30 years of their life, if it's to be truly general. It would also need bias correction, and maybe 2-5 other AGIs to form a consensus before making any decision properly. And even that would not be enough to fit in with humans - if you want an AGI to make the objectively right decisions, it will have to ignore many human feelings/biases. If you want it to fit in with people, it will have to make "mistakes", i.e. ineffective decisions.
- denzil_correa 7y agoYes, that’s true. But human decisions have accountability - algorithms don’t have accountability. If entities who deploy algorithms would be legally required to take accountability for algorithmic decisions, there would be no issue with black box algorithms.
- brycesbeard 7y agoI agree in spirit, but a lot of human behaviors, derived from decisions, don’t have consequences. Did Nestle face consequences for their decisions to promote formula, etc?
- denzil_correa 7y agoLet's not confuse accountability of decision making entity with consequences faced by the accountability. I was referring to the former, not the latter. The former is a case for law, the latter is a subject of law enforcement. In the case of Nestlé, you have a clear assigned accountable entity. We don't have that with algorithmic decision making (yet) where organizations wash off their hands saying "Oops! It was an algorithm, we didn't do anything!" Weak law enforcement is not a reason to not have proper laws in place.
- jotm 7y ago> We don't have that with algorithmic decision making (yet) where organizations wash off their hands saying "Oops! It was an algorithm, we didn't do anything!" That's what individuals within companies do all the time. You're onto something, form a limited liability company for your AIs and hire them as contractors. There, a clear assigned accountable entity :)
- wongarsu 7y agoWhy would organizations not be responsible for decisions made by an algorithm? After all they are still the entity who decided to use that algorithm, and to execute the decision. Sure, they say "Oops! It was an algorithm, we didn't do anything!", but is that any different than management saying "Oops! It was just a rogue employee, we didn't do anything!" For anything sufficiently consequential or systematic the second excuse doesn't work, so why should the first excuse work?
- jlawson 7y ago>how do we ensure machine learning is trained and verified in ways that don't encode bias Or, more commonly, people work to ensure their machine learning is trained and limited to not specifically notice any patterns of fact that would be politically unacceptable to notice and make use of, even if it was rational and useful for the task at hand. The status quo moral system of our society rests on a collection of objectively false beliefs about the physical world. Most humans know enough to avoid noticing these facts because of the social cost and possible societal negative impacts of shaking those moral foundations. It's not even necessarily wrong, since agreed-upon lies can do a lot to reduce conflict and cruelty. But machines need to be specifically trained to not notice these facts. You need to teach the machine not to notice the emperor's nudity. And what's even better is that this'll never get acknowledged in the literature. It's one of those self-hiding facts which exists if you're willing to notice it yourself, but which no authority will ever tell you (I love these, wish there was a word for them).
- gvjddbnvdrbv 7y agoIf we were able to provide perfect datasets you would be correct. As it is our datasets are imperfect and ML models are only as good as the training data.
- lukemerrick 7y agoThis is an incredibly salient point. Others have also pointed out that there are numerous applications in which black box models appear to offer significantly greater accuracy than interpretable models, bolstering the notion that this article is a bit overstated. However, in this article and elsewhere Professor Rudin has cited compelling evidence of cases in which black box models have been demonstrated to be no more accurate than interpretable alternatives. I feel this fairly justifies the question in the title of the article. For example, based upon available evidence, it appears reasonable that some onus should lie on the creators and buyers of COMPAS (a proprietary black box recidivism model) to demonstrate COMPAS actually is more accurate than an interpretable baseline. While it may not be the case, as the article seems to suggest, that in all modeling cases there is an interpretable alternative with comparable accuracy, in cases which there is, there doesn't seem to be any justification for using a black box model. On the matter of "human-style" interpretability, we are brought to the difference between "interpretability" and "explainability." Humans have a complex capacity for constructing explanations for the thoughts and actions of ourselves and others (among other things). As OP points out, a lot of famous psychological experiments by Kahneman and others have shown how much of our reasoning appears to be post-hoc, often biased, and often inaccurate (in other words, human explanations are not actually true transparent interpretations of our thoughts and actions). However, we humans do have a powerful capacity to evaluate and challenge the explanations presented by others, and we are able to reject bad explanations. For those interested, a great book on this topic is "The Enigma of Reason" by Mercier and Sperber (https://www.hup.harvard.edu/catalog.php?isbn=9780674237827 https://www.hup.harvard.edu/catalog.php?isbn=9780674237827), but the gist here is that we must understand that while explanations are not the same as transparent interpretability, they are still useful. I would conjecture that at some level of complexity (which some predictive tasks like pixel-to-label image recognition seem to exhibit), true end-to-end interpretability is not possible -- the best we can do is to construct an explanation. However, two very important points should be observed when considering this conjecture: 1. (Professor Rudin's point in the article) In cases which are not too complex for interpretable models to achieve comparable accuracy to black-box models, we can and should use them, as they offer super-human transparency at no cost in accuracy. 2. Constructing no explanations (or bad explanations) is not the same as reaching the same level of semi-transparency that humans offer. If we want to use human interpretability as a benchmark, black box models with no explanations are not up to par.
- sullyj3 7y agoBlack boxness isn't a property of the thing, it's a property of our understanding of the thing. Just because we don't understand the how pattern recognition in humans works, doesn't mean the process has no internal structure, or that we couldn't potentially understand it in the future.
- IshKebab 7y agoRight, and currently we understand neither human vision, nor CNN vision - they are both black boxes. I don't think anyone said they must remain black boxes forever. Edit: I guess you were referring to this: > Robust pattern recognition based on 100+'s of factors is just inherently black box. I think he meant " ... given our current knowledge."
- shrimpx 7y agoIt's true that humans are black boxes but there are whitebox techniques such as the scientific method and algorithms that are justified step by step. I think the article tries to compare black box learned algorithms to explicit algorithms. Humans being a blackbox is more of a philosophical issue from that perspective.
- tdhttt 7y agoOne more benefit: an interpretable model could at least give some hint when the model is not working as intended (e.g. debugging).
- choeger 7y agoSometimes I have the feeling that this is exactly why so many people prefer black-box modeling. You are not going to be criticised and you don't have to do all the hard work of software development.
- deleted 7y ago[deleted]
- agra 7y agoWas the fact that objects fall down a "black box" before Newton came up with some equations? I don't think its fair to describe CNN's and human vision as being black boxes in the same way.
- wongarsu 7y agoThe fact that objects fall down was well understood, obviously. What wasn't understood was the mechanism that makes this happen. Why did that apple fall down instead of up? Where exactly is down a round earth? Where is down when I stand next to the Himalaya? Why don't planets fall down? In my eyes this is very similar to our lack of understanding about the inner workings of a CNN, or human vision. We understand the input and the output and their rough relationship, we just don't understand how the move from input to output happens.
- agra 7y agoI see it differently. The distinction I would make is that even before Newton, people could make useful predictions about the trajectory of falling objects. My dog does it well, every time he catches a ball. He even seems to grok the effect of wind on a frisbee. We have no way of predicting how a CNN will react to an input. GANs really illustrate this point. I think this is an important distinction.
- zajio1am 7y ago> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction. Does it matter? If a brain counstructs post-hoc argument for given decision and the argument is defensible against other arguments, then the argument itself is good enough substantiation of the decision, even if it is different from the actual decision process.
- 6gvONxR4sf7o 7y agoNot all human thinking is the same. How you decide the amount to tip your waiter is not at all the same as how your doctor diagnoses you. And at we have a zillion years of evolution towards understanding and judging and conversing with each other's black boxes.