12 ms·

Scientists Increasingly Can’t Explain How AI Works

145 points by isaacfrond 4y ago

thinkmcfly 4y ago
What do you know, more sensationalized journalism from vice
Ragnarork 4y ago
> Scientists do not attempt to decipher the “black box,” or the opaque processes that the system undertakes, as long as they receive the outputs they are looking for. This seems a bit disingenuous as there's a whole research field (explainability) that does exactly that. This is not the widest of the whole machine learning domain, but it's there and there's significant work around it...
rcarr 4y ago
Is this true or is it more a case that journalists increasingly can’t understand scientists?
motohagiography 4y ago
Now do reproducable builds. This is the same complexity problem, but with a bunch of woo as a vehicle for their other agenda items. I could write the same article by saying, "developers increasingly can't explain how software works," and then use the ancient aliens conclusion-from-negative-premise trope to say, "software is everywhere and complex, scientists can't even reproduce it, <interview with scientist saying reproducable builds and complete code path coverage in testing are hard>, could software contain embedded problematic and systemic bads like sexism, racism and homophobia? These legitimate PhD's say they don't know for sure. We asked further simple questions, but they were ignored, could software developers be hiding other problematic things? - ancient critical theorists say 'yes.'" I'm not saying it's stupid, but I am saying it's as honest and intellectually rigorous as Ancient Aliens.
amai 4y ago
Nobody can explain how AI works, because it is a bullshit buzzword invented by people who don't understand how computers work. It's like asking how artificial horses work. It's just not a useful question to ask.
rifty 4y ago
I do not think there is always going to be a practical accompanying strategy for all things weighted impressions can return results for. ‘Intuition’ at times being extremely more practical isn’t a problem to me. But perhaps this is a weird concept for some people who see themselves fundamentally as a machine of reason; that the concept of intuition is simply a result of ignorance rather than a alternative viable method of process… The important thing to me is that it is predictable, just like universal phenomena we cannot not explain where it comes from, yet can prove through repeated experiment to be a law; or moldable to command like with the behaviour of a paint brush we learn to master with our hand.
cashsterling 4y ago
My 0.02... I'm not a neural network expert but have done a fair amount of nonlinear dynamic modeling and large-system optimization work in my career. I have always felt (and commented on HH) that neural network based approaches to complex problems would work well up to a point and then fundamentally not be able to go any further because of the massive degrees of freedom in NN-based modeling and the emergent erratic, non-linear behavior. So a specific NN model could get me 95% accuracy or "performance" on some task, but 98%, or 99.5%, is mathematically impossible... limit of "impossibility" being different for each problem. Trained neural networks of reasonable size have many thousands of independent variables and, as a result, have incredibly complex dynamics and behavior. Conversely, this explains why neural networks (of sometimes modest size) are able to approximately model behavior of extremely complex systems, linear or non-linear. We can often understand the sensitivity of the NN to a specific variable in the NN, or the interplay of three variables together. Once we need to consider tens or hundreds of variables, we really can't fathom the dynamics and interplay of variables in the model any more... in some cases, iterative numerical sensitivity analysis can help. But even reasonably small NN's can have millions of variables and these numerical analysis methods become intractable to apply. Remember the news of about 'single pixel' attacks against machine learning models that analyze and "categorize" images... researcher could change the category of an image by changing one pixel, the "right pixel", or hot pink or something. The researchers where in effect exploring a small sub-space of the NN and finding "regions of extreme dynamics" (highly weird behavior) in the model. There is no "rationale" for changing a cat to a dog based on one pixel going hot-pink in a 2M pixel... but there are extreme dynamics in model that allow that to happen. I worked for a time on infrared detector development and "badly behaving" pixels (blinkers, etc.) were enemy #1 for IR computer vision because the wrong blinking pixel in a 4M+ pixel imaging array could cause all kinds of problems in the CV algorithms. It was okay for a pixel to be dead, or hot, or have low quantum efficiency... but pixels that blink, flicker, etc. periodically caused extremely erratic, unpredictable behavior of video analysis algorithms, which were largely NN-based, or used other high-degree of freedom modeling / decision-making methods. While we can't predict this kind of weird behavior ahead of time from a specific NN model of reasonable size, we expect it. As briefly discussed above, we can't inspect complex NN models and figure out how and where they are going wrong. We'd have to perform an exhaustive, permutation-based search of the NN dynamic 'phase space' to determine what variable tweaks in the NN achieve the affects we are looking for... that search is, I think, a NP-hard problem (but I'm not sure). There are a lot of other factors influencing the general "hardness" of modeling complex things with neural networks (data quality, etc.)... but this idea of "complex dynamics" in NN's is, I think, the fundamental problem that can be partially mitigated but not completely solved.
bartimus 4y ago
Building the system that understands how the AI is coming to certain results/conclusion is probably way more complicated than the AI itself.
brk 4y ago
The overly simplified explanation is that the AI models see patterns in the training data that we do not. Feed a system a series of pictures of stop signs, and humans might key on things like the red color, or the shape, or even the word "STOP". An AI model might also recognize that they commonly have 2 bolts, and are attached to a post, and have a contrasting border. Typical machine vision systems can only classify what they have been trained on, they are ignorant to other objects. Train a system on a bunch of STOP signs, and then show it a SPEED LIMIT sign with morning sun giving the sign an amber glow, and the AI system might call it a STOP sign. Again, this is an overly simplified explanation, but we can explain how AI works, we just can't always explain or know what parameters in the training data it keyed on.
harryf 4y ago
> AI experts and researchers are warning developers to take a step back and focus more on how and why a system produces certain results than the fact that the system can accurately and rapidly produce them. We need to find a way to incentivize that otherwise it’s not going to happen. Perhaps looking how the security industry works with “buy bounties” is a place to borrow ideas from?
- Avicebron 4y ago
  I fairly familiar with bug bounties in the security industry, what do you propose specifically?
croes 4y ago
To me the problem of AI is not how it works but how reliable it works. Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?
- CuriouslyC 4y ago
  To be fair, this is a problem with people too. We use "interpersonal weirdness" as a proxy for "may produce catastrophic results" to some degree.
- ben_w 4y ago
  We can't even do that with humans. Best we've got is qualifications, and punishments for breaking rules.
- dmichulke 4y ago
  What you're saying is (kinda) my response when people ask me about self-driving cars: How does the car deal with object X on the road? For X=piano, a stack of solar panels, a tank, an airplane, a pile of stones ... No one knows and since the size of the set of X is infinite, no one can appropriately train for it either. That's why we can't have self-driving cars without a general understanding of what objects are and how they move.
  - pixelfarmer 4y ago
    Yes and no. Yes, because I agree with what you said. No, because it assumes self-driving has to rely purely on normal vision/cameras, which is just garbage². Think LiDAR and other such means to detect obstacles, but things like that are (currently) pretty expensive, so there is the attempt to just do it with (more or less) normal cameras, which leads to what you said (and I fully agree with that). Aside of the fact that safety assumptions+validations towards cameras are complicated in itself already, not only the technology in all its aspects, but also dirty lenses and things like that.
    - williamcotton 4y ago
      LiDAR can't tell you mass of an object as it is headed towards your vehicle. Imagine you're driving down the road and you see: 1.) A kid throw a rubber playground ball into the road and at your windshield. 2.) A kid heave a bowling ball into the road and at your windshield. How would you react in each case? How do you distinguish between the two? As a kid is heaving the bowling ball you can tell that it has a lot more mass by how the kid had to move his body to throw the ball. You learned that because you spent the first few years of your life experiencing the basic physics of reality. You've internalized the motion of the human body. LiDAR can also have a dirty lens!
Xcelerate 4y ago
There is a somewhat widespread implicit assumption that having sufficient intelligence to develop increasingly sophisticated machine learning implies that we also have sufficient intelligence to develop an “intuitive” understanding of how it works. I think that assumption is totally wrong, and trying to reconcile the two is probably a distraction and a waste of time. Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises? With regard to the moral questions around machine learning, I think people are overthinking things. There needs to be a sharp (societal) line between inferring causation and the capability for prediction. Maximize the latter any way that you can, but a lot of careful thought should be put into how the results of a prediction are utilized. Instead we are currently going about this backward by trying to haphazardly “clean” the input data so that the output gets a free pass on how it can be used. We can never decouple all of the biases and eliminate intra-predictivity in the input data though, so it’s fundamentally a bad approach.
- xg15 4y ago
  > There is a somewhat widespread implicit assumption that having sufficient intelligence to develop increasingly sophisticated machine learning implies that we also have sufficient intelligence to develop an “intuitive” understanding of how it works. I agree with you on that point, but not with the conclusion that this is in any way a good thing. > Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises? I think I have the complete opposite view here. Why would "being the catalyst in the next stage of evolution" be anything desirable? Evolution has no "stages", it's just chaos and life trying to deal in the best way with the circumstances at hand. If we blew everything up in nuclear armageddon tomorrow and the only remaining life were archaeae, those would be a perfectly fine "next stage of evolution". Similarly, we can build a would in which only robots can survive, but why would we want to? I think, gaining more understanding how the world works is the one thing AI can really bring to the table. If it solves practical problems on the way to that, that's great, but we evidently survived so far pretty well by doing that stuff ourselves.
- fallingknife 4y ago
FartyMcFarter 4y ago
> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has. I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers. In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. But this is no better than explaining the output of a neural network as "matrix multiplication plus a few non-linearities". Even back then a chess expert could try and rationalize certain moves in human-like terms: "oh, Deep Blue realised it needs to fight for the dark squares on the queenside". But this wasn't a real explanation, just like "this part of the picture looks like dog hair" isn't necessarily a correct explanation for why an AI labels an image as a dog. Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.
- xg15 4y ago
  How would you debug such a system? E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug? Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question. Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour? I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.
  - FartyMcFarter 4y ago
    > E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned. Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").
bobosha 4y ago
I think explainability is overrated (to use a Trumpian expression) & this fixation with coming up with explanations for inferences is a red herring. We cannot explain our own thoughts and actions and tend to ascribe logic & reason to many of our own actions, but it's almost always system-1 driven, for the most part. Correcting for edge-cases and unknown-unknowns is where we should focus our efforts methinks.
- endorphinbomber 4y ago
  You can't explain your own thoughts and actions??? I'm sorry, but that's part of makes a valuable team mate or partner. This is not fundamentally human, this is trained behaviour. I am very well capable of expressing my thougts and explaining actions resulting from these thoughts.
  - kuramitropolis 4y ago
    Can you explain why you had a thought, however? And how many levels deep? Only as many as someone cares to ask, I'd wager. You have developed the capability of explaining your actions as resulting from your thoughts, because there is someone on the other side who is ready to accept and evaluate your explanations. Others have no such affordance, so their brains get trained in prejudices and superstitions indead. Even if an explanation of an action is given in good faith and accepted by the recipient, it can still be factually incorrect.
- mdp2021 4y ago
  A very critical core human activity is that of reflection. You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
  - bobosha 4y ago
    Thanks, but enough neuroscience studies have shown that our decisions are for the most part, on autopilot, qv: Kahneman, Tversky et al. All our "explanations" are post-hoc rationalization i.e. reality is the narrative we tell ourselves. >You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear. Our learning is also for the most part Hebbian, from childhood through adulthood. For example, a DUI might force one to rethink their transport after a night-out. Right now, some of the "shocking" predictions have a child-like brutal honesty about them. Just as a child is coached not to call that bad aunty "fatty", an AI can thus be trained to conform to societal norms. tl;dr: i am advocating for a realtime continuous learning system
joxel 4y ago
It's quite easy to explain. You take a bunch of tensors and multiply them by a bunch of tensors. Humans can't wrap their heads around multiple tensors being multiplied together and never will. It's not a problem with AI. It's a problem with humans. It's not AIs fault that we can understand F=ma but can't understand 50 tensors being stacked.
- djmips 4y ago
  That's like saying you can explain how some code works by saying you take a bunch of transistors and turn them off and on.
- datpiff 4y ago
  >It's not a problem with AI. It's a problem with humans. It's not AIs fault This is a really, really funny defense of AI.
  - joxel 4y ago
    Why? How is it AI's fault that it can solve problems humans have proven unable to solve? And then you think it's AI's fault when we can't explain how it's solving the problems we can't solve? There's a reason we can't explain why a neural network detects it as a cat or dog, it's because we can't solve it ourselves, so the exact mathematical intuition is literally beyond our ability to describe (so far).
    - aliqot 4y ago
      "How can a thousand monkeys solve what humans can't solve"
      - joxel 4y ago
        That is not analogous
        deleted 4y ago
        [deleted]
photochemsyn 4y ago
The history of scientific development is one of finding patterns in data (think Kepler's studies of Tycho Brahe's accurate astronomical observations) - yes, AI excels at that - but then expressing those patterns in terms of simple mathematical equations: orbits are elliptical to a first approximation (not counting other graviational bodies than the sun and a planet), and the sun->planet vector sweeps out equal areas in equal times, etc. https://www.physicsclassroom.com/class/circles/Lesson-4/Kepler-s-Three-Laws https://www.physicsclassroom.com/class/circles/Lesson-4/Kepl... AI doesn't seem to excel at producing such foundational relationships that can then be used as building blocks in more complex theories. That is, you could train an AI on Tycho Brahe's observations and get it to predict the future positions of the planets with high accuracy, but would it spit out Kepler's laws?
- mdp2021 4y ago
  > yes, AI excels at that Sorry for being pedantic about the terminology, but AI has been "deterministic", using direct algorithms as opposed to oracular machines, since those times in which the perceptron was relatively weak. What we are talking about here is very probably Artificial Neural Networks.
- sebzim4500 4y ago
  True but the vast majority of humans couldn't do that either.
- chii 4y ago
  but what if it spat out einsteins equations, and that it somehow magically works but the people looking at that output could not imagine the "explanation"? Wouldn't that look exactly like what AI we have today - able to make patterns or predictinos, but we cannot interpret the AI's "formula".
  - AstralStorm 4y ago
    It should at least attempt to simplify its theorem, if even mechanically.
Simon_O_Rourke 4y ago
And people think this is alarming? I know responsible folks who can't explain how their critical enterprise APIs work.
fallingknife 4y ago
Previous technology has observed things that people can't by essentially increasing the power of one of our senses, e.g. a microscope increases visual resolution, or a radio telescope increases visual range. This is very easy for the brain to understand. But if you can build a machine to improve on the human brain itself in ability to recognize patterns, is it even possible to get the brain to understand it?
hoseja 4y ago
Only a fourth paragraph before the lede is revealed: racial and gender biases!
- mdp2021 4y ago
  I cannot imagine your reaction when you will arrive at the part where a pressure will be mentioned to «change "blacklist" to "blocklist"». Well, Andrew Tanenbaum remembered when at IBM he received a full explanation of why they felt very important his shirt should not just be of some specific colour, but of the specific shade of some colour. I would not say it is not part of the job: I would say it ["we feel it very important"] is part of "what happens".
  - ogogmad 4y ago
    > it's worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example Attention, you peasants! Our overlords have decided for us that we may no longer say blackbox and whitebox. They have given us the following alternatives. You must all choose: - Glassbox vs magicbox - Openbox vs closedbox - A global find/replace on the word "black", because that might be easier at this point
    - deleted 4y ago
      [deleted]
saladtoes 4y ago
Explainability is not a given in many more traditional complex systems. Decisions are often an aggregation of a large number of signals, and one can often not conceive of a single intuitive explanation for the system's decisions. A lot is expected of AI systems today, from fairness (how do we even define that?) to universality. In my view we need to develop a practical understanding of what it means to build the system we have in mind: do I understand where I want my system to perform, and do I have the tools to assess whether I am getting there? Interpretability is orthogonal to all of this. I would much rather have a well tested system, accompanied by online monitoring to detect unusual inputs in an ever-changing data distribution and notify when updates are needed or a human needs to take control, than an unreliable system that is great at providing explanations.
- mdp2021 4y ago
  > I would much rather have False dichotomy. In fact, well understood systems must be more reliable.
  - saladtoes 4y ago
    Is that really a universal fact? In any case, my statement goes in the opposite direction: is a reliable system necessarily "well understood", in the sense that it can explain its decisions? Most complex systems powering our lives cannot tell us anything about how they made those decisions. AI systems add a layer of complexity. Even if you can explain a decision well on your training data, I seriously doubt you will be able to still provide reasonable explanations in completely out of distribution data.
recuter 4y ago
> Black box models can be extremely powerful, which is how many scientists and companies justify sacrificing explainability for accuracy. > AI systems have been used for autonomous cars, customer service chatbots, and diagnosing disease, and have the power to perform some tasks better than humans can. For example, a machine that is capable of remembering one trillion items, such as digits, letters, and words, versus humans, who on average remember seven in their short-term memory would be able to process and compute information at a much faster and improved rate than humans. I suspect this article was written by AI. Or a hack journalist. And most of you haven't read it and are reacting to the title. Certainly won't find any explanations of how AI works on vice.
- ClumsyPilot 4y ago
  > suspect this article was written by AI. Or a hack journalist. A friend of mine did not belive that the AI based astroturfing problem is real, he though he would tell apart human tweets / letters / phonecalls vs bots. But have you seen some of the dumb shit real people post? the overlap between mental people and bad ai is huge
  - dwringer 4y ago
    I'm not sure how many people have messed around with the Character.ai beta but personally I'm awed that it does some of the things it does. It predictably fails at times and makes some odd mistakes, but at other times it insightfully points things out that I hadn't considered and explains itself quite eloquently when prompted. It's a lot better discussing literature and philosophy than science, though... and a compulsive liar. But a human editor willing to paper over the inconsistencies can definitely churn out articles in a fraction of the time with language models like this. And in regular conversation it's easily up there with, or surpasses, your average internet troll (which, as you pointed out, isn't exactly a high bar).
    - htrp 4y ago
      We're also providing the input to train character ai on good quality results
klik99 4y ago
I guess the title “Scientists Increasingly Running Out Of Ways To Explain AI To Reporters That Won’t Lead To Completely Misleading Articles” just didn’t have enough punch
kriro 4y ago
I don't see a problem with not being able to explain how something works as long as it's not failing empirically. Scientists also cannot explain how humans work in detail and yet humans are "allowed" to do many things and make many decisions that cannot be fully explained. The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? In fact it is ethically problematic (at least for me) to not use a medical AI that outperforms doctors due to liability issues or claims of not being certain how it works. Oftentimes the only question is "what if something goes horribly wrong" but th hidden cost of letting humans do the task at 85% instead of some AI that could do it at 90% is rarely considered.
- omginternets 4y ago
  There's the obvious problem of knowing if and when it might suddenly start failing empirically.
  - science4sail 4y ago
    Isn't that also a problem with humans? A human pilot might suddenly go insane and start murdering people (even though empirically, it's an improbable event). https://en.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.wikipedia.org/wiki/Germanwings_Flight_9525
    - omginternets 4y ago
      If your point is just that machines can be more reliable than humans, that's not a terribly interesting argument. Humans are often held accountable for the behavior of the machines they oversee. This is no exception.
- Verdex 4y ago
  I'm mostly in agreement.[1] The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine. We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality. And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind. [1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.) However, even so, I would not allow such a doctor to roam free. Perhaps irrationally. The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy. For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.
mabbo 4y ago
Scientists increasingly can't explain how any intelligence works. Look, I threw some photons from a screen to your eyes. How the heck are you reading this? What is going on? Well, if you can't explain it, that must mean it's bad, right?
philipov 4y ago
AI works the way human intuition does. Try explaining professional intuition to someone; it's not going to be a convincing argument unless they're willing to trust your expert gut feeling.
kybernetyk 4y ago
Who would have guessed that having a highly complex black box means you can't explain what's going on inside that black box... My pet theory is that this is the reason why Siri, Alexa and co. still are shit and haven't moved an inch forward since their inception. I like to play "sleep music" through Alexa when I bring my kid to bed. For his mid day nap it all works perfectly. But in the evening when I say the same phrase that worked a few hours before the thing first plays "dance charts" and when I repeat myself it plays "german rap" - if I repeat myself again I'm back to "dance charts". This happens only in the evening. Next day for daytime nap it all works as expected. Siri has the quirk that it won't turn off lights in the evening. During day it's all fine. In the evening "Siri turn off lights in the living room" ... nope. It fails. Every time. So I have to manually open the Home app and turn off the lights. (Turning on the lights via Siri works- turning off not). Have fun debugging these, Apple and Amazon.
- alcover 4y ago
  > "Siri turn off lights in the living room" No snark here : why don't you make a few steps and switch them off manually ? Saving these few steps doesn't make sense to me. You need to walk to be healthy.
  - mynameisvlad 4y ago
    "Siri turn off all the lights" is a hell of a lot easier than going up and down the stairs to every room in my house to make sure I turned off all lights before going to bed.
  - kybernetyk 4y ago
    Because the Hue lights aren't connected to a switch :) It's just accent lighting that's all over the room.
- gcanyon 4y ago
  Siri has gotten better in at least one measurable way: since it first came out, I've repeatedly cited this phrase as a problem: "Hey Siri, text this photo to <person's name>" For years Siri would respond with "I'm sorry, I don't have the ability to send photos." Specifically I would point out that Siri understood what I wanted, it just couldn't do it. Each time there is an OS update, I check that to see if it's been addressed, and I've been disappointed for years. Recently in a job interview in response to one of those "what's a product you like or don't like and why" questions, I said that I liked Siri, but here's my concrete example of how it needs to improve. I pulled out my phone, said the phrase... and Siri happily texted the photo. I didn't pass the interview :-)
jonnycomputer 4y ago
I used to believe that if you can build it you must understand it. But as soon as I started learning about evolutionary algorithms (e.g. for circuit design), I realized that building it would only be the first step in actually understanding it.
mdp2021 4y ago
Now for a small elephant which entered the room: the article recites that it would be > worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example Very plainly, the idea of "black-box" comes from the clear, basic and original, notion and experience that "in the dark, you cannot see". So, back to the point: we need transparency in AI, because we need insight as much as we can gather, because there appears to be a drought, an arificial scarcity, of good sense, of [un]common sense. We need every boost of good, [un]common sense very direly.
- whiddershins 4y ago
  Black box was actually a literal non-reflective black box inside an airplane that housed secret tech. https://en.wikipedia.org/wiki/Flight_recorder#:~:text=The%20term%20%22black%20box%22%20was,reflective%20black%20boxes%20or%20housings https://en.wikipedia.org/wiki/Flight_recorder#:~:text=The%20....
  - mdp2021 4y ago
    > actually Of course it was, but the expression stands because of the opacity implied by 'black'. The [original] box contains secrets - it may even be rigged -, or anyway very complex technology, state of the art, just developed or evolved, especially difficult to understand - you do not know how it works. It is black, and that "black" is significant in many ways. (And of course, a "flight recorder" is not the same thing: it is a specific function - not of the most esoteric - of the original pack of potential embedded technologies, which inherited the name. The radar, for example, becomes "mainstream", while the recorder remains "boxed" because you want it protected for "emergential witnessing".)
- belval 4y ago
  I guess it's the hill I'll die on, but these "white vs black" and "master slave" are inherently racist just make my head turn every time. blacklist came from BEFORE slavery in America. > According to the Henry Holt Encyclopedia of Word and Phrase Origins the word "blacklist" originated with a list England's King Charles II made of fifty-eight judges and court officers who sentenced his father, Charles I, to death in 1649. When Charles II was restored to the throne in 1660, thirteen of these regicides were executed and twenty-five sentenced to life imprisonment, while others escaped. Thinking that blackbox is racist is approaching insane level of mind-bending "everything is about race".
insane_dreamer 4y ago
If it could be explained, it wouldn't be a "black box" would it?
prego_xo 4y ago
We can always rely on Vice to find our dystopias.
deleted 4y ago
[deleted]
ivoras 4y ago
The armchair philosopher in me really wants to draw a parallel to a loose interpretation of Godel's incompleteness theorem to conclude that there's nothing which indicates we should be able to understand even us, let alone emergent phenomena we provoke.
- mdp2021 4y ago
  > The armchair philosopher in me I hope that the armchair pragmatist in you would react, when told "I would not trust your wife", by asking for more information.
rgllm 4y ago
We need an AI to explain AI.
dekhn 4y ago
We (scientists) greatly overestimate our ability to explain how we explain our decisions. Not sure why people think AI would somehow be more comprehensible. it seems to be a quirk of the human brain that there is a logical explainable reason (narrative).
moe091 4y ago
If we could explain exactly how it works wouldn't we just write some "normal" logic to do what it does instead of creating a NN and having it learn on it's own. Even AI from 40 years ago is probably impossible to explain completely
alex_suzuki 4y ago
Skynet… and so it begins.
cultofmetatron 4y ago
if the human brain was simple to understand, we wouldn't be smart enough to understand it. Is it really surprising that as AI approaches our level of intelligence, the underlying interactions become more and more of a black box?
tehsauce 4y ago
A system only being able to perform tasks that you understand is a huge limitation. This will only continue as civilization transitions to silicon based life.
habitue 4y ago
I think it's very possible that we won't have good explanations for things that are this complicated and that change this quickly. I.e. massive matrices of numbers that are trained and retrained regularly. Humans have a small amount of working memory, and anything bigger has to be knitted together from pointers to concepts stored in long term memory. Long term memory takes a while to construct. Let's say you take one single forward pass of a neutral network to output a single image. To really understand what's happening, you'd have to study that network for years, tracing which parts of the network's matrices influence which parts of the output image. How long would it take to really build a deep intuition about that single network, or even that single forward pass? I'd think as a starting point it would take as long to learn as a language, and probably much longer.
jjk166 4y ago
That's the point. If we had good and intuitive models of how a system works, we could simply write a program to calculate whatever result we were looking for. The purpose of AI is to look at data and find patterns that the human mind struggles to pick up thus allowing us to make accurate predictions without understanding the underlying rules. In the early days of AI when we were basically just practicing, we applied them to problems that we knew the answers to because we understood the underlying rules, and thus could see easily how close AIs got to accurately learning these rules. But now as AI has graduated to the level where it can be used for real world applications, of course now it is successfully doing its job of figuring out relations which are not only difficult to spot but also difficult to turn into an intuitive narrative.
- mdp2021 4y ago
  > The purpose of AI ... patterns No. That is just some model of AI. I think I should advise following the publicly available MIT course of late Prof. Patrick Winston. > as AI has graduated to the level where it can be used for real world applications I am pretty sure we had applications in the '50s. > it is successfully ... figuring out relations which are not only difficult to spot but also difficult to turn into an intuitive narrative And we would like to know them, for many reasons. Because in some cases what we are looking for is the full solution as opposed to the conclusion; because we do not just trust advice blindly; because it is productive in the very engineering effort...
  - jjk166 4y ago
    > No. That is just some model of AI. Finding patterns in data without being told explicitly what to look for is the hallmark of AI. Anything that doesn't do that isn't AI. > I am pretty sure we had applications in the '50s. For a different definition of applications than the one I was clearly using, yes. > And we would like to know them, for many reasons. Because in some cases what we are looking for is the full solution as opposed to the conclusion; because we do not just trust advice blindly; because it is productive in the very engineering effort... And I would like my car not to crash, but that's not the point of a seatbelt. Of course it would be wonderful if we understood everything, but we don't, and so we have created a tool to let us overcome that limitation.
kelseyfrog 4y ago
This is good actually. It means that AI is becoming increasingly more like the human brain - also something we cannot explain how it works. Were we under some sort of pretense that AI would be explainable? What's the ancestry of that idea?
- notahacker 4y ago
  Our aspirations for AI once included the assumption that intelligence involved the ability to explain its own thought processes... Settling for "computer programs with interesting output that don't know how they work but neither do we" seems like a step back. Many things are too complicated for us to parse without remotely approximating intelligence, like next month's weather.
  - kelseyfrog 4y ago
    > Our aspirations for AI once included the assumption that intelligence involved the ability to explain its own thought processes Any idea who this idea traces back to? It would seem they were rather off the mark in their predictions.
6d6b73 4y ago
In other words - ai doesnt work and nobody knows why

145 points by isaacfrond 4y ago

What do you know, more sensationalized journalism from vice

> Scientists do not attempt to decipher the “black box,” or the opaque processes that the system undertakes, as long as they receive the outputs they are looking for. This seems a bit disingenuous as there's a whole research field (explainability) that does exactly that. This is not the widest of the whole machine learning domain, but it's there and there's significant work around it...

rcarr 4y ago

Is this true or is it more a case that journalists increasingly can’t understand scientists?

motohagiography 4y ago

Now do reproducable builds. This is the same complexity problem, but with a bunch of woo as a vehicle for their other agenda items. I could write the same article by saying, "developers increasingly can't explain how software works," and then use the ancient aliens conclusion-from-negative-premise trope to say, "software is everywhere and complex, scientists can't even reproduce it, <interview with scientist saying reproducable builds and complete code path coverage in testing are hard>, could software contain embedded problematic and systemic bads like sexism, racism and homophobia? These legitimate PhD's say they don't know for sure. We asked further simple questions, but they were ignored, could software developers be hiding other problematic things? - ancient critical theorists say 'yes.'" I'm not saying it's stupid, but I am saying it's as honest and intellectually rigorous as Ancient Aliens.

amai 4y ago

Nobody can explain how AI works, because it is a bullshit buzzword invented by people who don't understand how computers work. It's like asking how artificial horses work. It's just not a useful question to ask.

rifty 4y ago

I do not think there is always going to be a practical accompanying strategy for all things weighted impressions can return results for. ‘Intuition’ at times being extremely more practical isn’t a problem to me. But perhaps this is a weird concept for some people who see themselves fundamentally as a machine of reason; that the concept of intuition is simply a result of ignorance rather than a alternative viable method of process… The important thing to me is that it is predictable, just like universal phenomena we cannot not explain where it comes from, yet can prove through repeated experiment to be a law; or moldable to command like with the behaviour of a paint brush we learn to master with our hand.

cashsterling 4y ago

My 0.02... I'm not a neural network expert but have done a fair amount of nonlinear dynamic modeling and large-system optimization work in my career. I have always felt (and commented on HH) that neural network based approaches to complex problems would work well up to a point and then fundamentally not be able to go any further because of the massive degrees of freedom in NN-based modeling and the emergent erratic, non-linear behavior. So a specific NN model could get me 95% accuracy or "performance" on some task, but 98%, or 99.5%, is mathematically impossible... limit of "impossibility" being different for each problem. Trained neural networks of reasonable size have many thousands of independent variables and, as a result, have incredibly complex dynamics and behavior. Conversely, this explains why neural networks (of sometimes modest size) are able to approximately model behavior of extremely complex systems, linear or non-linear. We can often understand the sensitivity of the NN to a specific variable in the NN, or the interplay of three variables together. Once we need to consider tens or hundreds of variables, we really can't fathom the dynamics and interplay of variables in the model any more... in some cases, iterative numerical sensitivity analysis can help. But even reasonably small NN's can have millions of variables and these numerical analysis methods become intractable to apply. Remember the news of about 'single pixel' attacks against machine learning models that analyze and "categorize" images... researcher could change the category of an image by changing one pixel, the "right pixel", or hot pink or something. The researchers where in effect exploring a small sub-space of the NN and finding "regions of extreme dynamics" (highly weird behavior) in the model. There is no "rationale" for changing a cat to a dog based on one pixel going hot-pink in a 2M pixel... but there are extreme dynamics in model that allow that to happen. I worked for a time on infrared detector development and "badly behaving" pixels (blinkers, etc.) were enemy #1 for IR computer vision because the wrong blinking pixel in a 4M+ pixel imaging array could cause all kinds of problems in the CV algorithms. It was okay for a pixel to be dead, or hot, or have low quantum efficiency... but pixels that blink, flicker, etc. periodically caused extremely erratic, unpredictable behavior of video analysis algorithms, which were largely NN-based, or used other high-degree of freedom modeling / decision-making methods. While we can't predict this kind of weird behavior ahead of time from a specific NN model of reasonable size, we expect it. As briefly discussed above, we can't inspect complex NN models and figure out how and where they are going wrong. We'd have to perform an exhaustive, permutation-based search of the NN dynamic 'phase space' to determine what variable tweaks in the NN achieve the affects we are looking for... that search is, I think, a NP-hard problem (but I'm not sure). There are a lot of other factors influencing the general "hardness" of modeling complex things with neural networks (data quality, etc.)... but this idea of "complex dynamics" in NN's is, I think, the fundamental problem that can be partially mitigated but not completely solved.

bartimus 4y ago

Building the system that understands how the AI is coming to certain results/conclusion is probably way more complicated than the AI itself.

brk 4y ago

The overly simplified explanation is that the AI models see patterns in the training data that we do not. Feed a system a series of pictures of stop signs, and humans might key on things like the red color, or the shape, or even the word "STOP". An AI model might also recognize that they commonly have 2 bolts, and are attached to a post, and have a contrasting border. Typical machine vision systems can only classify what they have been trained on, they are ignorant to other objects. Train a system on a bunch of STOP signs, and then show it a SPEED LIMIT sign with morning sun giving the sign an amber glow, and the AI system might call it a STOP sign. Again, this is an overly simplified explanation, but we can explain how AI works, we just can't always explain or know what parameters in the training data it keyed on.

harryf 4y ago

> AI experts and researchers are warning developers to take a step back and focus more on how and why a system produces certain results than the fact that the system can accurately and rapidly produce them. We need to find a way to incentivize that otherwise it’s not going to happen. Perhaps looking how the security industry works with “buy bounties” is a place to borrow ideas from?

Avicebron 4y ago
I fairly familiar with bug bounties in the security industry, what do you propose specifically?

croes 4y ago

To me the problem of AI is not how it works but how reliable it works. Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?

CuriouslyC 4y ago
To be fair, this is a problem with people too. We use "interpersonal weirdness" as a proxy for "may produce catastrophic results" to some degree.
ben_w 4y ago
We can't even do that with humans. Best we've got is qualifications, and punishments for breaking rules.
dmichulke 4y ago
What you're saying is (kinda) my response when people ask me about self-driving cars: How does the car deal with object X on the road? For X=piano, a stack of solar panels, a tank, an airplane, a pile of stones ... No one knows and since the size of the set of X is infinite, no one can appropriately train for it either. That's why we can't have self-driving cars without a general understanding of what objects are and how they move.
- pixelfarmer 4y ago
  Yes and no. Yes, because I agree with what you said. No, because it assumes self-driving has to rely purely on normal vision/cameras, which is just garbage². Think LiDAR and other such means to detect obstacles, but things like that are (currently) pretty expensive, so there is the attempt to just do it with (more or less) normal cameras, which leads to what you said (and I fully agree with that). Aside of the fact that safety assumptions+validations towards cameras are complicated in itself already, not only the technology in all its aspects, but also dirty lenses and things like that.
  - williamcotton 4y ago
    LiDAR can't tell you mass of an object as it is headed towards your vehicle. Imagine you're driving down the road and you see: 1.) A kid throw a rubber playground ball into the road and at your windshield. 2.) A kid heave a bowling ball into the road and at your windshield. How would you react in each case? How do you distinguish between the two? As a kid is heaving the bowling ball you can tell that it has a lot more mass by how the kid had to move his body to throw the ball. You learned that because you spent the first few years of your life experiencing the basic physics of reality. You've internalized the motion of the human body. LiDAR can also have a dirty lens!

Xcelerate 4y ago

There is a somewhat widespread implicit assumption that having sufficient intelligence to develop increasingly sophisticated machine learning implies that we also have sufficient intelligence to develop an “intuitive” understanding of how it works. I think that assumption is totally wrong, and trying to reconcile the two is probably a distraction and a waste of time. Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises? With regard to the moral questions around machine learning, I think people are overthinking things. There needs to be a sharp (societal) line between inferring causation and the capability for prediction. Maximize the latter any way that you can, but a lot of careful thought should be put into how the results of a prediction are utilized. Instead we are currently going about this backward by trying to haphazardly “clean” the input data so that the output gets a free pass on how it can be used. We can never decouple all of the biases and eliminate intra-predictivity in the input data though, so it’s fundamentally a bad approach.

xg15 4y ago
> There is a somewhat widespread implicit assumption that having sufficient intelligence to develop increasingly sophisticated machine learning implies that we also have sufficient intelligence to develop an “intuitive” understanding of how it works. I agree with you on that point, but not with the conclusion that this is in any way a good thing. > Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises? I think I have the complete opposite view here. Why would "being the catalyst in the next stage of evolution" be anything desirable? Evolution has no "stages", it's just chaos and life trying to deal in the best way with the circumstances at hand. If we blew everything up in nuclear armageddon tomorrow and the only remaining life were archaeae, those would be a perfectly fine "next stage of evolution". Similarly, we can build a would in which only robots can survive, but why would we want to? I think, gaining more understanding how the world works is the one thing AI can really bring to the table. If it solves practical problems on the way to that, that's great, but we evidently survived so far pretty well by doing that stuff ourselves.
fallingknife 4y ago

FartyMcFarter 4y ago

> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has. I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers. In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. But this is no better than explaining the output of a neural network as "matrix multiplication plus a few non-linearities". Even back then a chess expert could try and rationalize certain moves in human-like terms: "oh, Deep Blue realised it needs to fight for the dark squares on the queenside". But this wasn't a real explanation, just like "this part of the picture looks like dog hair" isn't necessarily a correct explanation for why an AI labels an image as a dog. Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.

xg15 4y ago
How would you debug such a system? E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug? Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question. Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour? I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.
- FartyMcFarter 4y ago
  > E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned. Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").

bobosha 4y ago

I think explainability is overrated (to use a Trumpian expression) & this fixation with coming up with explanations for inferences is a red herring. We cannot explain our own thoughts and actions and tend to ascribe logic & reason to many of our own actions, but it's almost always system-1 driven, for the most part. Correcting for edge-cases and unknown-unknowns is where we should focus our efforts methinks.

endorphinbomber 4y ago
You can't explain your own thoughts and actions??? I'm sorry, but that's part of makes a valuable team mate or partner. This is not fundamentally human, this is trained behaviour. I am very well capable of expressing my thougts and explaining actions resulting from these thoughts.
- kuramitropolis 4y ago
  Can you explain why you had a thought, however? And how many levels deep? Only as many as someone cares to ask, I'd wager. You have developed the capability of explaining your actions as resulting from your thoughts, because there is someone on the other side who is ready to accept and evaluate your explanations. Others have no such affordance, so their brains get trained in prejudices and superstitions indead. Even if an explanation of an action is given in good faith and accepted by the recipient, it can still be factually incorrect.
mdp2021 4y ago
A very critical core human activity is that of reflection. You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
- bobosha 4y ago
  Thanks, but enough neuroscience studies have shown that our decisions are for the most part, on autopilot, qv: Kahneman, Tversky et al. All our "explanations" are post-hoc rationalization i.e. reality is the narrative we tell ourselves. >You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear. Our learning is also for the most part Hebbian, from childhood through adulthood. For example, a DUI might force one to rethink their transport after a night-out. Right now, some of the "shocking" predictions have a child-like brutal honesty about them. Just as a child is coached not to call that bad aunty "fatty", an AI can thus be trained to conform to societal norms. tl;dr: i am advocating for a realtime continuous learning system

joxel 4y ago

It's quite easy to explain. You take a bunch of tensors and multiply them by a bunch of tensors. Humans can't wrap their heads around multiple tensors being multiplied together and never will. It's not a problem with AI. It's a problem with humans. It's not AIs fault that we can understand F=ma but can't understand 50 tensors being stacked.

djmips 4y ago
That's like saying you can explain how some code works by saying you take a bunch of transistors and turn them off and on.
datpiff 4y ago
>It's not a problem with AI. It's a problem with humans. It's not AIs fault This is a really, really funny defense of AI.
- joxel 4y ago
  Why? How is it AI's fault that it can solve problems humans have proven unable to solve? And then you think it's AI's fault when we can't explain how it's solving the problems we can't solve? There's a reason we can't explain why a neural network detects it as a cat or dog, it's because we can't solve it ourselves, so the exact mathematical intuition is literally beyond our ability to describe (so far).
  - aliqot 4y ago
    "How can a thousand monkeys solve what humans can't solve"
    - joxel 4y ago
      That is not analogous
      - deleted 4y ago
        [deleted]

photochemsyn 4y ago

The history of scientific development is one of finding patterns in data (think Kepler's studies of Tycho Brahe's accurate astronomical observations) - yes, AI excels at that - but then expressing those patterns in terms of simple mathematical equations: orbits are elliptical to a first approximation (not counting other graviational bodies than the sun and a planet), and the sun->planet vector sweeps out equal areas in equal times, etc. https://www.physicsclassroom.com/class/circles/Lesson-4/Kepler-s-Three-Laws https://www.physicsclassroom.com/class/circles/Lesson-4/Kepl... AI doesn't seem to excel at producing such foundational relationships that can then be used as building blocks in more complex theories. That is, you could train an AI on Tycho Brahe's observations and get it to predict the future positions of the planets with high accuracy, but would it spit out Kepler's laws?

mdp2021 4y ago
> yes, AI excels at that Sorry for being pedantic about the terminology, but AI has been "deterministic", using direct algorithms as opposed to oracular machines, since those times in which the perceptron was relatively weak. What we are talking about here is very probably Artificial Neural Networks.
sebzim4500 4y ago
True but the vast majority of humans couldn't do that either.
chii 4y ago
but what if it spat out einsteins equations, and that it somehow magically works but the people looking at that output could not imagine the "explanation"? Wouldn't that look exactly like what AI we have today - able to make patterns or predictinos, but we cannot interpret the AI's "formula".
- AstralStorm 4y ago
  It should at least attempt to simplify its theorem, if even mechanically.

Simon_O_Rourke 4y ago

And people think this is alarming? I know responsible folks who can't explain how their critical enterprise APIs work.

fallingknife 4y ago

Previous technology has observed things that people can't by essentially increasing the power of one of our senses, e.g. a microscope increases visual resolution, or a radio telescope increases visual range. This is very easy for the brain to understand. But if you can build a machine to improve on the human brain itself in ability to recognize patterns, is it even possible to get the brain to understand it?

hoseja 4y ago

Only a fourth paragraph before the lede is revealed: racial and gender biases!

mdp2021 4y ago
I cannot imagine your reaction when you will arrive at the part where a pressure will be mentioned to «change "blacklist" to "blocklist"». Well, Andrew Tanenbaum remembered when at IBM he received a full explanation of why they felt very important his shirt should not just be of some specific colour, but of the specific shade of some colour. I would not say it is not part of the job: I would say it ["we feel it very important"] is part of "what happens".
- ogogmad 4y ago
  > it's worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example Attention, you peasants! Our overlords have decided for us that we may no longer say blackbox and whitebox. They have given us the following alternatives. You must all choose: - Glassbox vs magicbox - Openbox vs closedbox - A global find/replace on the word "black", because that might be easier at this point
  - deleted 4y ago
    [deleted]

saladtoes 4y ago

Explainability is not a given in many more traditional complex systems. Decisions are often an aggregation of a large number of signals, and one can often not conceive of a single intuitive explanation for the system's decisions. A lot is expected of AI systems today, from fairness (how do we even define that?) to universality. In my view we need to develop a practical understanding of what it means to build the system we have in mind: do I understand where I want my system to perform, and do I have the tools to assess whether I am getting there? Interpretability is orthogonal to all of this. I would much rather have a well tested system, accompanied by online monitoring to detect unusual inputs in an ever-changing data distribution and notify when updates are needed or a human needs to take control, than an unreliable system that is great at providing explanations.

mdp2021 4y ago
> I would much rather have False dichotomy. In fact, well understood systems must be more reliable.
- saladtoes 4y ago
  Is that really a universal fact? In any case, my statement goes in the opposite direction: is a reliable system necessarily "well understood", in the sense that it can explain its decisions? Most complex systems powering our lives cannot tell us anything about how they made those decisions. AI systems add a layer of complexity. Even if you can explain a decision well on your training data, I seriously doubt you will be able to still provide reasonable explanations in completely out of distribution data.

recuter 4y ago

> Black box models can be extremely powerful, which is how many scientists and companies justify sacrificing explainability for accuracy. > AI systems have been used for autonomous cars, customer service chatbots, and diagnosing disease, and have the power to perform some tasks better than humans can. For example, a machine that is capable of remembering one trillion items, such as digits, letters, and words, versus humans, who on average remember seven in their short-term memory would be able to process and compute information at a much faster and improved rate than humans. I suspect this article was written by AI. Or a hack journalist. And most of you haven't read it and are reacting to the title. Certainly won't find any explanations of how AI works on vice.

ClumsyPilot 4y ago
> suspect this article was written by AI. Or a hack journalist. A friend of mine did not belive that the AI based astroturfing problem is real, he though he would tell apart human tweets / letters / phonecalls vs bots. But have you seen some of the dumb shit real people post? the overlap between mental people and bad ai is huge
- dwringer 4y ago
  I'm not sure how many people have messed around with the Character.ai beta but personally I'm awed that it does some of the things it does. It predictably fails at times and makes some odd mistakes, but at other times it insightfully points things out that I hadn't considered and explains itself quite eloquently when prompted. It's a lot better discussing literature and philosophy than science, though... and a compulsive liar. But a human editor willing to paper over the inconsistencies can definitely churn out articles in a fraction of the time with language models like this. And in regular conversation it's easily up there with, or surpasses, your average internet troll (which, as you pointed out, isn't exactly a high bar).
  - htrp 4y ago
    We're also providing the input to train character ai on good quality results

klik99 4y ago

I guess the title “Scientists Increasingly Running Out Of Ways To Explain AI To Reporters That Won’t Lead To Completely Misleading Articles” just didn’t have enough punch

kriro 4y ago

I don't see a problem with not being able to explain how something works as long as it's not failing empirically. Scientists also cannot explain how humans work in detail and yet humans are "allowed" to do many things and make many decisions that cannot be fully explained. The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? In fact it is ethically problematic (at least for me) to not use a medical AI that outperforms doctors due to liability issues or claims of not being certain how it works. Oftentimes the only question is "what if something goes horribly wrong" but th hidden cost of letting humans do the task at 85% instead of some AI that could do it at 90% is rarely considered.

omginternets 4y ago
There's the obvious problem of knowing if and when it might suddenly start failing empirically.
- science4sail 4y ago
  Isn't that also a problem with humans? A human pilot might suddenly go insane and start murdering people (even though empirically, it's an improbable event). https://en.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.wikipedia.org/wiki/Germanwings_Flight_9525
  - omginternets 4y ago
    If your point is just that machines can be more reliable than humans, that's not a terribly interesting argument. Humans are often held accountable for the behavior of the machines they oversee. This is no exception.
Verdex 4y ago
I'm mostly in agreement.[1] The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine. We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality. And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind. [1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.) However, even so, I would not allow such a doctor to roam free. Perhaps irrationally. The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy. For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.

mabbo 4y ago

Scientists increasingly can't explain how any intelligence works. Look, I threw some photons from a screen to your eyes. How the heck are you reading this? What is going on? Well, if you can't explain it, that must mean it's bad, right?

philipov 4y ago

AI works the way human intuition does. Try explaining professional intuition to someone; it's not going to be a convincing argument unless they're willing to trust your expert gut feeling.

kybernetyk 4y ago

Who would have guessed that having a highly complex black box means you can't explain what's going on inside that black box... My pet theory is that this is the reason why Siri, Alexa and co. still are shit and haven't moved an inch forward since their inception. I like to play "sleep music" through Alexa when I bring my kid to bed. For his mid day nap it all works perfectly. But in the evening when I say the same phrase that worked a few hours before the thing first plays "dance charts" and when I repeat myself it plays "german rap" - if I repeat myself again I'm back to "dance charts". This happens only in the evening. Next day for daytime nap it all works as expected. Siri has the quirk that it won't turn off lights in the evening. During day it's all fine. In the evening "Siri turn off lights in the living room" ... nope. It fails. Every time. So I have to manually open the Home app and turn off the lights. (Turning on the lights via Siri works- turning off not). Have fun debugging these, Apple and Amazon.

alcover 4y ago
> "Siri turn off lights in the living room" No snark here : why don't you make a few steps and switch them off manually ? Saving these few steps doesn't make sense to me. You need to walk to be healthy.
- mynameisvlad 4y ago
  "Siri turn off all the lights" is a hell of a lot easier than going up and down the stairs to every room in my house to make sure I turned off all lights before going to bed.
- kybernetyk 4y ago
  Because the Hue lights aren't connected to a switch :) It's just accent lighting that's all over the room.
gcanyon 4y ago
Siri has gotten better in at least one measurable way: since it first came out, I've repeatedly cited this phrase as a problem: "Hey Siri, text this photo to <person's name>" For years Siri would respond with "I'm sorry, I don't have the ability to send photos." Specifically I would point out that Siri understood what I wanted, it just couldn't do it. Each time there is an OS update, I check that to see if it's been addressed, and I've been disappointed for years. Recently in a job interview in response to one of those "what's a product you like or don't like and why" questions, I said that I liked Siri, but here's my concrete example of how it needs to improve. I pulled out my phone, said the phrase... and Siri happily texted the photo. I didn't pass the interview :-)

jonnycomputer 4y ago

I used to believe that if you can build it you must understand it. But as soon as I started learning about evolutionary algorithms (e.g. for circuit design), I realized that building it would only be the first step in actually understanding it.

mdp2021 4y ago

Now for a small elephant which entered the room: the article recites that it would be > worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example Very plainly, the idea of "black-box" comes from the clear, basic and original, notion and experience that "in the dark, you cannot see". So, back to the point: we need transparency in AI, because we need insight as much as we can gather, because there appears to be a drought, an arificial scarcity, of good sense, of [un]common sense. We need every boost of good, [un]common sense very direly.

whiddershins 4y ago
Black box was actually a literal non-reflective black box inside an airplane that housed secret tech. https://en.wikipedia.org/wiki/Flight_recorder#:~:text=The%20term%20%22black%20box%22%20was,reflective%20black%20boxes%20or%20housings https://en.wikipedia.org/wiki/Flight_recorder#:~:text=The%20....
- mdp2021 4y ago
  > actually Of course it was, but the expression stands because of the opacity implied by 'black'. The [original] box contains secrets - it may even be rigged -, or anyway very complex technology, state of the art, just developed or evolved, especially difficult to understand - you do not know how it works. It is black, and that "black" is significant in many ways. (And of course, a "flight recorder" is not the same thing: it is a specific function - not of the most esoteric - of the original pack of potential embedded technologies, which inherited the name. The radar, for example, becomes "mainstream", while the recorder remains "boxed" because you want it protected for "emergential witnessing".)
belval 4y ago
I guess it's the hill I'll die on, but these "white vs black" and "master slave" are inherently racist just make my head turn every time. blacklist came from BEFORE slavery in America. > According to the Henry Holt Encyclopedia of Word and Phrase Origins the word "blacklist" originated with a list England's King Charles II made of fifty-eight judges and court officers who sentenced his father, Charles I, to death in 1649. When Charles II was restored to the throne in 1660, thirteen of these regicides were executed and twenty-five sentenced to life imprisonment, while others escaped. Thinking that blackbox is racist is approaching insane level of mind-bending "everything is about race".

insane_dreamer 4y ago

If it could be explained, it wouldn't be a "black box" would it?

prego_xo 4y ago

We can always rely on Vice to find our dystopias.

deleted 4y ago

[deleted]

ivoras 4y ago

The armchair philosopher in me really wants to draw a parallel to a loose interpretation of Godel's incompleteness theorem to conclude that there's nothing which indicates we should be able to understand even us, let alone emergent phenomena we provoke.

mdp2021 4y ago
> The armchair philosopher in me I hope that the armchair pragmatist in you would react, when told "I would not trust your wife", by asking for more information.

rgllm 4y ago

We need an AI to explain AI.

dekhn 4y ago

We (scientists) greatly overestimate our ability to explain how we explain our decisions. Not sure why people think AI would somehow be more comprehensible. it seems to be a quirk of the human brain that there is a logical explainable reason (narrative).

moe091 4y ago

If we could explain exactly how it works wouldn't we just write some "normal" logic to do what it does instead of creating a NN and having it learn on it's own. Even AI from 40 years ago is probably impossible to explain completely

alex_suzuki 4y ago

Skynet… and so it begins.

cultofmetatron 4y ago

if the human brain was simple to understand, we wouldn't be smart enough to understand it. Is it really surprising that as AI approaches our level of intelligence, the underlying interactions become more and more of a black box?

tehsauce 4y ago

A system only being able to perform tasks that you understand is a huge limitation. This will only continue as civilization transitions to silicon based life.

habitue 4y ago

I think it's very possible that we won't have good explanations for things that are this complicated and that change this quickly. I.e. massive matrices of numbers that are trained and retrained regularly. Humans have a small amount of working memory, and anything bigger has to be knitted together from pointers to concepts stored in long term memory. Long term memory takes a while to construct. Let's say you take one single forward pass of a neutral network to output a single image. To really understand what's happening, you'd have to study that network for years, tracing which parts of the network's matrices influence which parts of the output image. How long would it take to really build a deep intuition about that single network, or even that single forward pass? I'd think as a starting point it would take as long to learn as a language, and probably much longer.

jjk166 4y ago

That's the point. If we had good and intuitive models of how a system works, we could simply write a program to calculate whatever result we were looking for. The purpose of AI is to look at data and find patterns that the human mind struggles to pick up thus allowing us to make accurate predictions without understanding the underlying rules. In the early days of AI when we were basically just practicing, we applied them to problems that we knew the answers to because we understood the underlying rules, and thus could see easily how close AIs got to accurately learning these rules. But now as AI has graduated to the level where it can be used for real world applications, of course now it is successfully doing its job of figuring out relations which are not only difficult to spot but also difficult to turn into an intuitive narrative.

mdp2021 4y ago
> The purpose of AI ... patterns No. That is just some model of AI. I think I should advise following the publicly available MIT course of late Prof. Patrick Winston. > as AI has graduated to the level where it can be used for real world applications I am pretty sure we had applications in the '50s. > it is successfully ... figuring out relations which are not only difficult to spot but also difficult to turn into an intuitive narrative And we would like to know them, for many reasons. Because in some cases what we are looking for is the full solution as opposed to the conclusion; because we do not just trust advice blindly; because it is productive in the very engineering effort...
- jjk166 4y ago
  > No. That is just some model of AI. Finding patterns in data without being told explicitly what to look for is the hallmark of AI. Anything that doesn't do that isn't AI. > I am pretty sure we had applications in the '50s. For a different definition of applications than the one I was clearly using, yes. > And we would like to know them, for many reasons. Because in some cases what we are looking for is the full solution as opposed to the conclusion; because we do not just trust advice blindly; because it is productive in the very engineering effort... And I would like my car not to crash, but that's not the point of a seatbelt. Of course it would be wonderful if we understood everything, but we don't, and so we have created a tool to let us overcome that limitation.

kelseyfrog 4y ago

This is good actually. It means that AI is becoming increasingly more like the human brain - also something we cannot explain how it works. Were we under some sort of pretense that AI would be explainable? What's the ancestry of that idea?

notahacker 4y ago
Our aspirations for AI once included the assumption that intelligence involved the ability to explain its own thought processes... Settling for "computer programs with interesting output that don't know how they work but neither do we" seems like a step back. Many things are too complicated for us to parse without remotely approximating intelligence, like next month's weather.
- kelseyfrog 4y ago
  > Our aspirations for AI once included the assumption that intelligence involved the ability to explain its own thought processes Any idea who this idea traces back to? It would seem they were rather off the mark in their predictions.

6d6b73 4y ago

In other words - ai doesnt work and nobody knows why