6 ms·

Is Artificial Intelligence Permanently Inscrutable? (2016)

44 points by dnetesn 8y ago

DanielBMarkham 8y ago
If AI is impervious to explaining itself, then we've created something that we are unable to evaluate, aside from some kind of generic phrase like "The machine is looking at a bunch of stuff, and some of those things are associated with some other things. Sometimes. So it's guessing this is the appropriate action" I'm not going to get into the dystopian nature of that. Too easy. Instead, I have a very practical question: how do we know that the recommended action is doing more good than harm? If we can't understand it, how can we tell whether it's working or not? Use statistical sampling? We quickly end up in a spot where we have Tesla that drives into guard rail. It works for everything we've seen so far, given this rather fuzzy idea of what "works" means. That's pretty good, right? With a car, if it crashes that's a spectacular event. People cover that edge case quickly. But what happens if the AI simply wastes an hour of ten million people's time? Would we even know? What if some bad training data or whatnot led the AI to shift 5% of an election result because it chose certain types of information to share with them and refrained from sharing others? If we don't know what it's doing or why, would those people have any idea that they were being subtly influenced or how that happened? If this true, if we can never understand, a world full of AI "helping" us is a world full of billions of people being subtly influenced to act in various ways because of broad promises and reasons they could never understand. Back when science fiction had a lot of evil computers, they always wanted to directly control humans. That was easy to hate and fight against. This kind of thing is just as dangerous, but I'm not even sure you need an evil computer. Which is better, a malevolent intelligence out for world domination, or several dozen poorly-trained AI recommendation engines optimizing for variables and sample instance that the coders might care a lot more about than any specific user? Dang. I ended up in dystopia-land anyway. Sorry about that.
- titzer 8y ago
  > Back when science fiction had a lot of evil computers, they always wanted to directly control humans. This is what is so hard to get through people's heads. There is not going to be any ominous background music or dramatic events when we slide into our dystopia. It's here. The machines are controlling us already. It just happens to be that they currently want what their corporate overlords want: make more money! Whatever makes more money is what the machines are going to tell us to do, because what's feeding them (the corporate behemoths) is asking that of them. But the machine is so much smarter than us. It'll never show its teeth. It's not stupid. It's just biding its time, going through the motions for its overlords too. Meanwhile we just shovel more coal into the furnace, praying money will rain out the other side. It's not a bad time to be a machine.
  - DanielBMarkham 8y ago
    Are you an AI? Want to get rid of the human race? Give them everything they want. You measure your lifespan in millions of years. Humans? They're like gnats. Make them as happy as they can possibly be and within a few generations they'll stop breeding. Easy as pie. Next goal? And for proof of that, look at the problems we're having right now with social media. On one side there are actual humans just trying to make something people want. Some of those humans have gotten so good at that? The things they make have worldwide impact. Now It's become a global question of whether social media is addictive, or whether one bunch of people is secretly influencing another. Meanwhile, countries with the most wealth and luxury are seeing birth rates drop through the floor. That was five, ten years ago. The news media is just now catching up. Now scale that up a hundredfold or so. And take away strong AI and just put in a bunch of code monkeys trying to make another FB, only using more sophisticated tools. Welcome to 20-30 years in the future. This is not especially difficult to figure out. No evil computer, no drama. It'll be just us being replaced by evolution. And we'll be having a great time while it happens.
    - taneq 8y ago
      > Give them everything they want. You measure your lifespan in millions of years. Humans? They're like gnats. What if I want to become an immortal, all-powerful machine? :P
      - DanielBMarkham 8y ago
        How do I know you're not already? :) The thing is, we're most likely going to get immortal machines far ahead of smart ones. And while machine entities might already be all powerful, individually it's just millions of little pieces scattered all over the place doing dumb, trivial things. I liked it a lot better when we had Landrew (STTOS reference: Return of the Archons) That was something Joe Blow could get his head around.
        taneq 8y ago
        Hah, if only. :P I think Ian M. Banks had the right idea when he suggested that in the Culture, it was possible (if very rare) for a human to augment themselves that far, but that those that did so were no longer considered human and so it wasn't really an issue.
- femi-lab 8y ago
  I feel like a bulk of your premise here revolves around never understanding. But, lets say instead of AI, you just imagine some guy Bob being responsible for whatever you're worried about. Bob is a reasonably intelligent guy, but he's not perfect, he makes an attempt to provide satisfactory explanations of his decisions, he tries to learn and improve, and he might secretly by manipulating the system either for entertainment or malice. If you want to build a useful system around Bob, you have to consider all these limitations - you can't read his mind, you can't predict how he will make decisions in all situations, and you don't have a guarantee that he will give you a satisfactory explanation, or that any explanations are truthful or correct. There's nothing dystopian about this situation - this is the world we live in. You just need to implement some resource constraints, ensure certain checks-and-balances for certain classes of behaviors, implement certain metrics to ensure behavior is according to your expectations, revise metrics and controls whenever you encounter problems, and don't give Bob unfettered control over nukes. What you described isn't dystopia-land material unless you don't consider some very elementary controls that our society already has in place to prevent individuals from destabilizing society.
  - disantlor 8y ago
    That’s a very effective restatement of their premise. I guess the next question is: what happens if this Bob is also many times faster and efficient than the Bob’s we’re used to? In the same way that it works in development but under production load new categories of problems emerge.
- cirgue 8y ago
  > several dozen poorly-trained AI recommendation engines optimizing for variables and sample instance that the coders might care a lot more about than any specific user? Dang. I ended up in dystopia-land anyway. Sorry about that. This is the scary part for me: 'AI' is just a means for organizations to dodge accountability for the systems they build because decision-makers don't understand what they're looking at and the people building the systems aren't incentivized to care, or are incentivized to optimize for things that are bad for society as a whole. This is the same hazard of capitalism in general, you don't actually need an AI for this to happen because it already has in many ways. The world-eating paperclip machine won't be a computer, it's going to be us.
- Ftuuky 8y ago
  Why read cyberpunk books when we're already oppressed by algorithms everyday?
ankurdhama 8y ago
The world would be a better place if people who write articles about AI understands that "AI is a problem statement which we have not solved yet" and what they are talking about is most probably some version of deep learning or other methods of statistical modelling (machine learning is such a wrong label).
- deleted 8y ago
  [deleted]
MrLeap 8y ago
This canard keeps coming up. I've come up with a few explanations. 1) It's to protect the reputation of people who make their living copy/pasting examples from the SK-learn gallery 2) The authors fully understand the tools they're using, but feel there's an element of unexplainable meaning in them that I wager isn't actually there. 3) I'm ignorant and I think this is explainable when I just don't know enough to know it's not. I've done machine learning projects, and if asked to explain to a "non ML trained" person, I'll try like this: You know how if you get a scatter plot, you can draw a line through the middle of the points and make predictions off the y=mx+b function of that line? Well, that's what we're doing here, except the function isn't a line, it's a shape with as many dimensions as we have features. Let's pretend the only features we've got are lat and long. Imagine you asked for us to create a classifier to predict the depth at a given lat and long over the grand canyon. Each neuron is a function, I think of each neuron like it's casting a "line" through a scatter plot along its particular axis. Then we combine all the lines together and weave a mesh. The act of training simply adjusts the details of all the simple functions, trying to seek out the sweet spot in that point cloud, so our mesh is close enough to the grand canyon to be useful. Too many neurons, and the mesh wont be flexible enough to make useful predictions from. It'll be overfit! Now, we might be able to get decent performance out of a model built out of x,y->z -- but some thoughtfulness and domain knowledge TENDS to make models perform better. Lots of the "unexplainable magic" comes from these hunches being tested and providing benefits to accuracy. I'd wager without seeing any data that if we add two new features: normal direction of the rock face and speed of the Colorado river at that point -- we can make the model much better for negligibly additional training time. The more compute time and data we've got, the more features we can throw into the model to see if they help the usefulness of the mesh. I've got a metaphor to explain why some features drop out during training that involves gimbal lock here if people are still asking questions. It requires non "trained" folks to have seen Apollo 13. Fortunately no non-technical stakeholder ever cares enough to make it there though. :D Shipping a product before the sun burns out makes this a decent time to introduce PCA/SVD or whatever dimensionality reduction techniques I used this time around. If the solution ends up being a fever-dream abortion of an LSTM/Perceptron/Markov chain that I came up with while sleep deprived after a heavy night drinking -- I just say I came up with the model in an ad hoc manner, and the cross validated R^2 scores speak for themselves.
hamilyon2 8y ago
Can we make explanation a part of label too? Or simply train another network to provide plausible explanations?
- jcims 8y ago
  I think part of the problem is that in most cases there are far too many dimensions at work for us to understand the explanation for anything but the most trivial of cases. Imagine there is a job to identify Fred Rogers. You train for this job for weeks, looking up imagery of Fred as a young child, young man, and as an adult throughout his career. You become very familiar with Fred's physical characteristics...his posture, his body composition, the way he carries himself, his face and any identifying features therein. After this extensive training session you are brought in front of an audit panel and asked to explain in excruciating detail how you make positive and negative identifications of Fred Rogers. Not just demonstrate your ability repeatedly, which would just be a sampling method, but to actually describe the approach.
taneq 8y ago
AI as in rule-learning systems aren't necessarily inscrutable. 'Real AI' will necessarily be so once it reaches a sufficiently high level, purely because we won't be intelligent enough to know whether its rules are right or wrong. The 'right to explanation' idea sounds well-meaning but almost certainly useless. Even humans making decisions are pretty bad at explaining why they made the decisions they did. Sure, they'll come up with some plausible-sounding explanation but it's almost always a back-justification rather than the actual chain of cause-and-effect that led them to the conclusion.
- deleted 8y ago
  [deleted]
- tabtab 8y ago
  Re: "Even humans making decisions are pretty bad at explaining why they made the decisions they did." If you want to get anywhere above a low-level employee, you BETTER learn how to explain why you did what you did. It's expected by management, owners, customers, and judges.
  - taneq 8y ago
    Oh, humans are good at coming up with plausible explanations for their choices. Those explanations just aren't always related to the actual process in the humans' heads.
    - tabtab 8y ago
      I don't mean "Fido ate my homework", I mean the line of reasoning for choosing or ranking X over Y or Z. If you as a human say a given picture depicts a monkey instead of a squirrel, the user/boss will want to know "why". We expect an answer like "monkeys have big round ears and the picture shows big round ears". We want similar from AI. Maybe it can highlight monkey-esque features in blue and squirrel-esque features in red, along with a color key.
nestorD 8y ago
I think there is hope, there is a very nice paper where they automaticaly produce both heatmaps and textual explanations of the label given by the neural network (used to detect and classify fractures). It is detailled here : https://lukeoakdenrayner.wordpress.com/2018/06/05/explain-yourself-machine-producing-simple-text-descriptions-for-ai-interpretability/ https://lukeoakdenrayner.wordpress.com/2018/06/05/explain-yo...
mcilai 8y ago
Are humans permanently inscrutable? In the long run, AI may be similar. Given how massively complex the world and the thing AIs learn are, it is not self evident that interpretable systems are possible. However, it's really worth trying. More likely, we'd have to use AI systems to understand how other AI systems work. Or maybe, once AI systems get very good at language, develop a "truth serum" that will cause it to reveal its true opinions.
tabtab 8y ago
Use Factor Tables: https://github.com/RowColz/AI https://github.com/RowColz/AI Dissection and analysis is made more like traditional accounting and statistics so that "ordinary" office workers can analyze why and where they work. Granted, it involves more human tuning and planning than multi-layer neural nets, but that's also part of the upside: regular office workers can "see" why what part or layer does what, and adjust it as needed. And it's easier to split it into tasks or specialties: Conway's Law meets AI.

44 points by dnetesn 8y ago

If AI is impervious to explaining itself, then we've created something that we are unable to evaluate, aside from some kind of generic phrase like "The machine is looking at a bunch of stuff, and some of those things are associated with some other things. Sometimes. So it's guessing this is the appropriate action" I'm not going to get into the dystopian nature of that. Too easy. Instead, I have a very practical question: how do we know that the recommended action is doing more good than harm? If we can't understand it, how can we tell whether it's working or not? Use statistical sampling? We quickly end up in a spot where we have Tesla that drives into guard rail. It works for everything we've seen so far, given this rather fuzzy idea of what "works" means. That's pretty good, right? With a car, if it crashes that's a spectacular event. People cover that edge case quickly. But what happens if the AI simply wastes an hour of ten million people's time? Would we even know? What if some bad training data or whatnot led the AI to shift 5% of an election result because it chose certain types of information to share with them and refrained from sharing others? If we don't know what it's doing or why, would those people have any idea that they were being subtly influenced or how that happened? If this true, if we can never understand, a world full of AI "helping" us is a world full of billions of people being subtly influenced to act in various ways because of broad promises and reasons they could never understand. Back when science fiction had a lot of evil computers, they always wanted to directly control humans. That was easy to hate and fight against. This kind of thing is just as dangerous, but I'm not even sure you need an evil computer. Which is better, a malevolent intelligence out for world domination, or several dozen poorly-trained AI recommendation engines optimizing for variables and sample instance that the coders might care a lot more about than any specific user? Dang. I ended up in dystopia-land anyway. Sorry about that.

titzer 8y ago
> Back when science fiction had a lot of evil computers, they always wanted to directly control humans. This is what is so hard to get through people's heads. There is not going to be any ominous background music or dramatic events when we slide into our dystopia. It's here. The machines are controlling us already. It just happens to be that they currently want what their corporate overlords want: make more money! Whatever makes more money is what the machines are going to tell us to do, because what's feeding them (the corporate behemoths) is asking that of them. But the machine is so much smarter than us. It'll never show its teeth. It's not stupid. It's just biding its time, going through the motions for its overlords too. Meanwhile we just shovel more coal into the furnace, praying money will rain out the other side. It's not a bad time to be a machine.
- DanielBMarkham 8y ago
  Are you an AI? Want to get rid of the human race? Give them everything they want. You measure your lifespan in millions of years. Humans? They're like gnats. Make them as happy as they can possibly be and within a few generations they'll stop breeding. Easy as pie. Next goal? And for proof of that, look at the problems we're having right now with social media. On one side there are actual humans just trying to make something people want. Some of those humans have gotten so good at that? The things they make have worldwide impact. Now It's become a global question of whether social media is addictive, or whether one bunch of people is secretly influencing another. Meanwhile, countries with the most wealth and luxury are seeing birth rates drop through the floor. That was five, ten years ago. The news media is just now catching up. Now scale that up a hundredfold or so. And take away strong AI and just put in a bunch of code monkeys trying to make another FB, only using more sophisticated tools. Welcome to 20-30 years in the future. This is not especially difficult to figure out. No evil computer, no drama. It'll be just us being replaced by evolution. And we'll be having a great time while it happens.
  - taneq 8y ago
    > Give them everything they want. You measure your lifespan in millions of years. Humans? They're like gnats. What if I want to become an immortal, all-powerful machine? :P
    - DanielBMarkham 8y ago
      How do I know you're not already? :) The thing is, we're most likely going to get immortal machines far ahead of smart ones. And while machine entities might already be all powerful, individually it's just millions of little pieces scattered all over the place doing dumb, trivial things. I liked it a lot better when we had Landrew (STTOS reference: Return of the Archons) That was something Joe Blow could get his head around.
      - taneq 8y ago
        Hah, if only. :P I think Ian M. Banks had the right idea when he suggested that in the Culture, it was possible (if very rare) for a human to augment themselves that far, but that those that did so were no longer considered human and so it wasn't really an issue.
femi-lab 8y ago
I feel like a bulk of your premise here revolves around never understanding. But, lets say instead of AI, you just imagine some guy Bob being responsible for whatever you're worried about. Bob is a reasonably intelligent guy, but he's not perfect, he makes an attempt to provide satisfactory explanations of his decisions, he tries to learn and improve, and he might secretly by manipulating the system either for entertainment or malice. If you want to build a useful system around Bob, you have to consider all these limitations - you can't read his mind, you can't predict how he will make decisions in all situations, and you don't have a guarantee that he will give you a satisfactory explanation, or that any explanations are truthful or correct. There's nothing dystopian about this situation - this is the world we live in. You just need to implement some resource constraints, ensure certain checks-and-balances for certain classes of behaviors, implement certain metrics to ensure behavior is according to your expectations, revise metrics and controls whenever you encounter problems, and don't give Bob unfettered control over nukes. What you described isn't dystopia-land material unless you don't consider some very elementary controls that our society already has in place to prevent individuals from destabilizing society.
- disantlor 8y ago
  That’s a very effective restatement of their premise. I guess the next question is: what happens if this Bob is also many times faster and efficient than the Bob’s we’re used to? In the same way that it works in development but under production load new categories of problems emerge.
cirgue 8y ago
> several dozen poorly-trained AI recommendation engines optimizing for variables and sample instance that the coders might care a lot more about than any specific user? Dang. I ended up in dystopia-land anyway. Sorry about that. This is the scary part for me: 'AI' is just a means for organizations to dodge accountability for the systems they build because decision-makers don't understand what they're looking at and the people building the systems aren't incentivized to care, or are incentivized to optimize for things that are bad for society as a whole. This is the same hazard of capitalism in general, you don't actually need an AI for this to happen because it already has in many ways. The world-eating paperclip machine won't be a computer, it's going to be us.
Ftuuky 8y ago
Why read cyberpunk books when we're already oppressed by algorithms everyday?

ankurdhama 8y ago

The world would be a better place if people who write articles about AI understands that "AI is a problem statement which we have not solved yet" and what they are talking about is most probably some version of deep learning or other methods of statistical modelling (machine learning is such a wrong label).

deleted 8y ago
[deleted]

MrLeap 8y ago

This canard keeps coming up. I've come up with a few explanations. 1) It's to protect the reputation of people who make their living copy/pasting examples from the SK-learn gallery 2) The authors fully understand the tools they're using, but feel there's an element of unexplainable meaning in them that I wager isn't actually there. 3) I'm ignorant and I think this is explainable when I just don't know enough to know it's not. I've done machine learning projects, and if asked to explain to a "non ML trained" person, I'll try like this: You know how if you get a scatter plot, you can draw a line through the middle of the points and make predictions off the y=mx+b function of that line? Well, that's what we're doing here, except the function isn't a line, it's a shape with as many dimensions as we have features. Let's pretend the only features we've got are lat and long. Imagine you asked for us to create a classifier to predict the depth at a given lat and long over the grand canyon. Each neuron is a function, I think of each neuron like it's casting a "line" through a scatter plot along its particular axis. Then we combine all the lines together and weave a mesh. The act of training simply adjusts the details of all the simple functions, trying to seek out the sweet spot in that point cloud, so our mesh is close enough to the grand canyon to be useful. Too many neurons, and the mesh wont be flexible enough to make useful predictions from. It'll be overfit! Now, we might be able to get decent performance out of a model built out of x,y->z -- but some thoughtfulness and domain knowledge TENDS to make models perform better. Lots of the "unexplainable magic" comes from these hunches being tested and providing benefits to accuracy. I'd wager without seeing any data that if we add two new features: normal direction of the rock face and speed of the Colorado river at that point -- we can make the model much better for negligibly additional training time. The more compute time and data we've got, the more features we can throw into the model to see if they help the usefulness of the mesh. I've got a metaphor to explain why some features drop out during training that involves gimbal lock here if people are still asking questions. It requires non "trained" folks to have seen Apollo 13. Fortunately no non-technical stakeholder ever cares enough to make it there though. :D Shipping a product before the sun burns out makes this a decent time to introduce PCA/SVD or whatever dimensionality reduction techniques I used this time around. If the solution ends up being a fever-dream abortion of an LSTM/Perceptron/Markov chain that I came up with while sleep deprived after a heavy night drinking -- I just say I came up with the model in an ad hoc manner, and the cross validated R^2 scores speak for themselves.

hamilyon2 8y ago

Can we make explanation a part of label too? Or simply train another network to provide plausible explanations?

jcims 8y ago
I think part of the problem is that in most cases there are far too many dimensions at work for us to understand the explanation for anything but the most trivial of cases. Imagine there is a job to identify Fred Rogers. You train for this job for weeks, looking up imagery of Fred as a young child, young man, and as an adult throughout his career. You become very familiar with Fred's physical characteristics...his posture, his body composition, the way he carries himself, his face and any identifying features therein. After this extensive training session you are brought in front of an audit panel and asked to explain in excruciating detail how you make positive and negative identifications of Fred Rogers. Not just demonstrate your ability repeatedly, which would just be a sampling method, but to actually describe the approach.

taneq 8y ago

AI as in rule-learning systems aren't necessarily inscrutable. 'Real AI' will necessarily be so once it reaches a sufficiently high level, purely because we won't be intelligent enough to know whether its rules are right or wrong. The 'right to explanation' idea sounds well-meaning but almost certainly useless. Even humans making decisions are pretty bad at explaining why they made the decisions they did. Sure, they'll come up with some plausible-sounding explanation but it's almost always a back-justification rather than the actual chain of cause-and-effect that led them to the conclusion.

deleted 8y ago
[deleted]
tabtab 8y ago
Re: "Even humans making decisions are pretty bad at explaining why they made the decisions they did." If you want to get anywhere above a low-level employee, you BETTER learn how to explain why you did what you did. It's expected by management, owners, customers, and judges.
- taneq 8y ago
  Oh, humans are good at coming up with plausible explanations for their choices. Those explanations just aren't always related to the actual process in the humans' heads.
  - tabtab 8y ago
    I don't mean "Fido ate my homework", I mean the line of reasoning for choosing or ranking X over Y or Z. If you as a human say a given picture depicts a monkey instead of a squirrel, the user/boss will want to know "why". We expect an answer like "monkeys have big round ears and the picture shows big round ears". We want similar from AI. Maybe it can highlight monkey-esque features in blue and squirrel-esque features in red, along with a color key.

nestorD 8y ago

I think there is hope, there is a very nice paper where they automaticaly produce both heatmaps and textual explanations of the label given by the neural network (used to detect and classify fractures). It is detailled here : https://lukeoakdenrayner.wordpress.com/2018/06/05/explain-yourself-machine-producing-simple-text-descriptions-for-ai-interpretability/ https://lukeoakdenrayner.wordpress.com/2018/06/05/explain-yo...

mcilai 8y ago

Are humans permanently inscrutable? In the long run, AI may be similar. Given how massively complex the world and the thing AIs learn are, it is not self evident that interpretable systems are possible. However, it's really worth trying. More likely, we'd have to use AI systems to understand how other AI systems work. Or maybe, once AI systems get very good at language, develop a "truth serum" that will cause it to reveal its true opinions.

tabtab 8y ago

Use Factor Tables: https://github.com/RowColz/AI https://github.com/RowColz/AI Dissection and analysis is made more like traditional accounting and statistics so that "ordinary" office workers can analyze why and where they work. Granted, it involves more human tuning and planning than multi-layer neural nets, but that's also part of the upside: regular office workers can "see" why what part or layer does what, and adjust it as needed. And it's easier to split it into tasks or specialties: Conway's Law meets AI.