8 ms·

Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achi

Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical. Also, people should have the right to know when machine-learned models are used to make decisions about their lives. They should be able to ask why a particular decisions was made and get that information. This is real AI ethics.

YeGoblynQueenne 7y ago
You know what AI systems had the ability to explain their decisions? Expert systems. Expert systems were the big success of GOFAI, but they fell out of favour in the last AI winter, at the end of the '90s or so, clearing the way for probabilistic inference and statistical machine learning. Since then it seems, we took one step forward (with accuracy in classification) and one step back (with the loss of the ability to explain decisions). Who knows, maybe a new AI winter will wipe out the statistical machine learning dinosaurs of today and leave a clear field of play for the AI Mammals of tomorrow.
- colllectorof 7y ago
  Well, there are some statistical learning models that are explainable. Decision trees can be induced and can "explain themselves", just like in expert systems. Additive models are self-explanatory. Not sure I'm sold on LIME and other similar approaches, though. Seems like a lot of deep learning people are all too happy to substitute "intepretability" for actual explanations.
  - YeGoblynQueenne 7y ago
    Decision trees are not a statistical model. The learning method is statistical, if you will, but the model itself is a disjunction of conjunctions- a symbolic model (and more specifically, a propositional logic theory). Decision Trees are in fact an example of the early years of machine learning where the trend was towards algorithms and techniques that learned symbolic theories. I believe the effort was driven by the realisation that expert systems had a certain problem with knowlege acquisition [1] which drove people to try and learn production rules from data. I digress- I mean to say that decision trees are explainable because their models are not statistical. To be honest, I don't know much about additive models. _______________ [1] https://en.wikipedia.org/wiki/Knowledge_acquisition https://en.wikipedia.org/wiki/Knowledge_acquisition
- nickpsecurity 7y ago
  Yup. That's what I learned first. Mycin being the first, success story I read about. https://en.wikipedia.org/wiki/Mycin https://en.wikipedia.org/wiki/Mycin I tried some quick DuckDuckGo-ing to see if anything new turned up. Drowned out by unrelated stuff. I did find what looks like a great, quick overview of expert systems for folks unfamiliar with them. Might be new, default link I share about how they historically were perceived. What you think of this one? https://www.tutorialspoint.com/artificial_intelligence/artificial_intelligence_expert_systems.htm https://www.tutorialspoint.com/artificial_intelligence/artif...
  - sgt101 7y ago
    The tutorial is a bit old. I think that advances in probablistic reasoning & modelling, such as practical Bayes networks should be included, and the mechanics of resolution have improved massively with the introduction of answerset systems - this gets over the problem of commitment that kiboshed gen 5. https://en.wikipedia.org/wiki/Answer_set_programming https://en.wikipedia.org/wiki/Answer_set_programming https://en.wikipedia.org/wiki/Probabilistic_programming_language https://en.wikipedia.org/wiki/Probabilistic_programming_lang...
- _bxg1 7y ago
  I work at this company: https://en.m.wikipedia.org/wiki/Cyc https://en.m.wikipedia.org/wiki/Cyc It's a survivor of the AI winter, and what we have is basically a generalized expert system. Every conclusion incorporates cross-domain knowledge and can fully explain itself. It's been a long, slow trudge of research and development over the past couple decades, but we're starting to poke our heads out and ride the current AI hype wave. We like to say that Watson is our marketing department.
  - YeGoblynQueenne 7y ago
    I know about Cyc, of course. Godspeed [and speed us away] :)
  - lawrenceyan 7y ago
    This seems like a fundamentally flawed and ultimately doomed approach to creating general agents. It's like you're trying to build a house, except only one molecule at a time.
    - _bxg1 7y ago
      The intent is to seed it by hand with enough knowledge that it will be able to do true NLU and learn new knowledge itself from the web. It's already shown some promise against these sorts of problems, which ML really struggles with: https://en.m.wikipedia.org/wiki/Winograd_Schema_Challenge https://en.m.wikipedia.org/wiki/Winograd_Schema_Challenge
      - nl 7y ago
        OpenAI's GPT-2 language model (build using a deep neural network) recently achieved 70.7% accuracy on the Winograd Schema Challenge, which is 8% better than the previous record[1]. I've never used Cyc but I have used OpenCyc and I'm familiar with some of the applications of Cyc. It's interesting when it works. [1] https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
      - lawrenceyan 7y ago
jl2718 7y ago
The dispute here is about gender classification. If you can give a consistent explanation of how your own visual system works for this task, then perhaps one day we’ll be able to program a computer to do the same.
- TeMPOraL 7y ago
  I can't, but human visual perception can be a subject to introspection to a high degree, and can be top-down overriden. That is, I can name at least some of the reasons why the person I'm looking at looks male/female to me, and in cases that are not obvious, I can step what I see through high-level, explicit reasoning, and the result actually informs my perception. Also, to the benefit of my perception works the fact that human beings have shared brain architecture, and the way I see stuff is the same as everybody else, so whatever half-assed explanation I could give, it's intimately understood by other people. In contrast, ML models are completely alien to us.
bko 7y ago
> Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical. What's the current standard for anything "important"? Most things important decisions are biased and not explained. Judges have biases and are allowed some discretion when sentencing. I'm sure police officers have biases as well. Same is true with just about any person making a judgement. There are laws (for good reason) that prevent certain types of biases, but it's naive to believe that the status quo on important decisions is great. A model is better in a number of ways. First it is based on actual evidence. Although that can be manipulated, it's a lot easier to observe control than the life experiences of an individual. And you can guarantee that certain factors won't play a direct role at least by excluding them as inputs. It's much harder to tell a person to ignore some factors. I think algorithmic, objective decision making on important decisions is very much preferable to what we have now.
- zwkrt 7y ago
  The word "objective" is problematic here. Normally people mean by this term "explainable by some quantitative criteria". In the case of AI this is technically true, but seems to lack substance. the workings of a neural networks for facial recognition could be decomposed to some god-awful switch statement involving all the pixels of the input picture (after all, this is exactly the structure of the resultant machine instructions), but how would one go about justifying any of the individual branches of such a structure? In other words, could you comment the code? I'm not 100% disagreeing with you, I just think it is problematic to call a statistical process "objective".
  - bko 7y ago
    Modern facial recognition uses landmarks, much like humans do. First it identifies a face, which is essentially a flat 2-d plane. It's able to identify locations of the nose, ears, mouth, chin, etc. It then looks for relationships between these features (e.g. how wide is the mouth, how far apart are the eyes, etc). It's a much simpler problem than the general image detection. Imagine how many different types of dogs you can encounter at various angles, sizes, colors, perspectives. For that it's much more of a black box as your model would have to support detecting a dog from behind, side, front, above, sitting down, walking, jumping, standing, dogs with long tails, short tails, long hair, short hair, etc. Now think about a face. Most faces are very much alike. https://github.com/ageitgey/face_recognition https://github.com/ageitgey/face_recognition
zwkrt 7y ago
This is such an interesting topic and I really don't know where to stand on it. On one hand, this is an argument that systems such as laws, lending practices, etc. should be based on repeatable and explainable rules. Rules will end up with some bias, but the bias can be dissected and argued about until we reach some consensus that the bias is acceptable/desiresble. In the case of lending, maybe we consider credit history a fair input metric but zip-code an unfair metric, for example. On the other hand, systems such as laws and lending, which are inherently about social interaction, do tend to have some unspecified human element. Judges interpret the law and apply it to specific cases, underwriters have latitude to make exceptions under certain vague circumstances, teachers may regrade a paper upon realizing they misspoke in a lecture. This is a feature, not a bug--if our social systems have no room for empathy then there is a big problem. So now that AI is "unexplainable", how is this worse than the unexplainability of human systems? You can ask a human why they took some action, but their explanation tends to be incomplete, wrong, a lie, or maybe they don't recall. I once was given the opportunity to lease a top-floor apartment in my building because I had a daily chat with the receptionist. All fair housing laws were followed, I just happened to be the first to know because I had such frequent interaction. If you were to ask the receptionist why I got the apartment she probably wouldn't say "zwkrt tells me bad jokes and we share pictures of our pets", but that probably is part of the reason. I think fundamentally we understand that humans share a way of life and a set of ineffable values, and we believe that computer-generated results do not share these values. Unfortunately I do not have a conclusion, just a formulation of a problem.
- atoav 7y ago
  The present systems have more attributes than “they somehow work, but slow”. Think about separation of power and decentralization. Slowness is a feature here. If you want to radically change a society where decisions are carried out by any number of individual actors, you need to take them with you by either using military force, money or convincing them otherwise. In any case this is a project that could also fail. With an algorithm, you could change it today and it would be in action without many even noticing it. I could get behind a simple and transparent tax system, where you just see in realtime what money you give while beeing sure that big company has to do the same, without gaming the system. But I am not sure the system that decides on these rules should be another system.
willvarfar 7y ago
Do you put the same requirement on natural neural nets? Is the decision reached by human members of a jury, for example, explainable in the way you mean?
Mirioron 7y ago
Can you explain how you, as a human being, understand handwriting? Can you explain it in detail without handwaving some stuff away as "now I detect the pattern"? Because I'm pretty sure you can't, but you use it for basically everything in life. So, should a person's eyes not be trusted when it comes to "something important"?
- afthonos 7y ago
  Given that eyewitness testimony is consistently unreliable, biased, and internally inconsistent, no, they shouldn’t. Human judgement is the best we can do so far, but it is hardly a worthwhile goal, especially with systems that take us from affecting individuals on a person-to-person level to influencing the interactions of millions of people with their government. Algorithms can be explainable, and I agree that anything that affects your standing in the eyes of government should have that as a minimum requirement.
- rizzin 7y ago
  >So, should a person's eyes not be trusted when it comes to "something important"? A documentary called "Murder on a Sunday Morning": https://www.youtube.com/watch?v=LFLbptkb1eM https://www.youtube.com/watch?v=LFLbptkb1eM A woman was murdered in front of her husband. The man saw the attacker up close and was the only eyewitness. He accused a completely innocent man, and it took half a year of jail and court-related turmoils for this to get cleared up.
- honzzz 7y ago
  > Can you explain how you, as a human being, understand handwriting? I do not want to speak for the parent but I think you might be on a different level of abstraction when talking about "explaining" things. Consider this for example: someone writes a letter to your boss that you should be fired. That someone does not need to explain the process of writing but probably should be required to explain why you should be fired.
- colllectorof 7y ago
  >Can you explain how you, as a human being, understand handwriting? We explain how to understand handwriting to every single child that goes to school. It's not the answer you want, but it's the answer that actually matters here. Trying to equivocate AI and human cognition in this way is completely disingenuous. Human reasoning is not 100% reliable, but we know very well in which ways it's unreliable and how to deal with it. We have shared biology and millennia of experience trying to empathize and communicate with others.
sireat 7y ago
The problem is that most machine learning model choices are humanly indecipherable. You can understand how the model works and the math behind it but you will be hard pressed to understand the exact path behind a particular choice. Decision trees are one exception to this. Most humans can understand those.

YeGoblynQueenne 7y ago

You know what AI systems had the ability to explain their decisions? Expert systems. Expert systems were the big success of GOFAI, but they fell out of favour in the last AI winter, at the end of the '90s or so, clearing the way for probabilistic inference and statistical machine learning. Since then it seems, we took one step forward (with accuracy in classification) and one step back (with the loss of the ability to explain decisions). Who knows, maybe a new AI winter will wipe out the statistical machine learning dinosaurs of today and leave a clear field of play for the AI Mammals of tomorrow.

colllectorof 7y ago
Well, there are some statistical learning models that are explainable. Decision trees can be induced and can "explain themselves", just like in expert systems. Additive models are self-explanatory. Not sure I'm sold on LIME and other similar approaches, though. Seems like a lot of deep learning people are all too happy to substitute "intepretability" for actual explanations.
- YeGoblynQueenne 7y ago
  Decision trees are not a statistical model. The learning method is statistical, if you will, but the model itself is a disjunction of conjunctions- a symbolic model (and more specifically, a propositional logic theory). Decision Trees are in fact an example of the early years of machine learning where the trend was towards algorithms and techniques that learned symbolic theories. I believe the effort was driven by the realisation that expert systems had a certain problem with knowlege acquisition [1] which drove people to try and learn production rules from data. I digress- I mean to say that decision trees are explainable because their models are not statistical. To be honest, I don't know much about additive models. _______________ [1] https://en.wikipedia.org/wiki/Knowledge_acquisition https://en.wikipedia.org/wiki/Knowledge_acquisition
nickpsecurity 7y ago
Yup. That's what I learned first. Mycin being the first, success story I read about. https://en.wikipedia.org/wiki/Mycin https://en.wikipedia.org/wiki/Mycin I tried some quick DuckDuckGo-ing to see if anything new turned up. Drowned out by unrelated stuff. I did find what looks like a great, quick overview of expert systems for folks unfamiliar with them. Might be new, default link I share about how they historically were perceived. What you think of this one? https://www.tutorialspoint.com/artificial_intelligence/artificial_intelligence_expert_systems.htm https://www.tutorialspoint.com/artificial_intelligence/artif...
- sgt101 7y ago
  The tutorial is a bit old. I think that advances in probablistic reasoning & modelling, such as practical Bayes networks should be included, and the mechanics of resolution have improved massively with the introduction of answerset systems - this gets over the problem of commitment that kiboshed gen 5. https://en.wikipedia.org/wiki/Answer_set_programming https://en.wikipedia.org/wiki/Answer_set_programming https://en.wikipedia.org/wiki/Probabilistic_programming_language https://en.wikipedia.org/wiki/Probabilistic_programming_lang...
_bxg1 7y ago
I work at this company: https://en.m.wikipedia.org/wiki/Cyc https://en.m.wikipedia.org/wiki/Cyc It's a survivor of the AI winter, and what we have is basically a generalized expert system. Every conclusion incorporates cross-domain knowledge and can fully explain itself. It's been a long, slow trudge of research and development over the past couple decades, but we're starting to poke our heads out and ride the current AI hype wave. We like to say that Watson is our marketing department.
- YeGoblynQueenne 7y ago
  I know about Cyc, of course. Godspeed [and speed us away] :)
- lawrenceyan 7y ago
  This seems like a fundamentally flawed and ultimately doomed approach to creating general agents. It's like you're trying to build a house, except only one molecule at a time.
  - _bxg1 7y ago
    The intent is to seed it by hand with enough knowledge that it will be able to do true NLU and learn new knowledge itself from the web. It's already shown some promise against these sorts of problems, which ML really struggles with: https://en.m.wikipedia.org/wiki/Winograd_Schema_Challenge https://en.m.wikipedia.org/wiki/Winograd_Schema_Challenge
    - nl 7y ago
      OpenAI's GPT-2 language model (build using a deep neural network) recently achieved 70.7% accuracy on the Winograd Schema Challenge, which is 8% better than the previous record[1]. I've never used Cyc but I have used OpenCyc and I'm familiar with some of the applications of Cyc. It's interesting when it works. [1] https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
    - lawrenceyan 7y ago

jl2718 7y ago

The dispute here is about gender classification. If you can give a consistent explanation of how your own visual system works for this task, then perhaps one day we’ll be able to program a computer to do the same.

TeMPOraL 7y ago
I can't, but human visual perception can be a subject to introspection to a high degree, and can be top-down overriden. That is, I can name at least some of the reasons why the person I'm looking at looks male/female to me, and in cases that are not obvious, I can step what I see through high-level, explicit reasoning, and the result actually informs my perception. Also, to the benefit of my perception works the fact that human beings have shared brain architecture, and the way I see stuff is the same as everybody else, so whatever half-assed explanation I could give, it's intimately understood by other people. In contrast, ML models are completely alien to us.

bko 7y ago

> Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical. What's the current standard for anything "important"? Most things important decisions are biased and not explained. Judges have biases and are allowed some discretion when sentencing. I'm sure police officers have biases as well. Same is true with just about any person making a judgement. There are laws (for good reason) that prevent certain types of biases, but it's naive to believe that the status quo on important decisions is great. A model is better in a number of ways. First it is based on actual evidence. Although that can be manipulated, it's a lot easier to observe control than the life experiences of an individual. And you can guarantee that certain factors won't play a direct role at least by excluding them as inputs. It's much harder to tell a person to ignore some factors. I think algorithmic, objective decision making on important decisions is very much preferable to what we have now.

zwkrt 7y ago
The word "objective" is problematic here. Normally people mean by this term "explainable by some quantitative criteria". In the case of AI this is technically true, but seems to lack substance. the workings of a neural networks for facial recognition could be decomposed to some god-awful switch statement involving all the pixels of the input picture (after all, this is exactly the structure of the resultant machine instructions), but how would one go about justifying any of the individual branches of such a structure? In other words, could you comment the code? I'm not 100% disagreeing with you, I just think it is problematic to call a statistical process "objective".
- bko 7y ago
  Modern facial recognition uses landmarks, much like humans do. First it identifies a face, which is essentially a flat 2-d plane. It's able to identify locations of the nose, ears, mouth, chin, etc. It then looks for relationships between these features (e.g. how wide is the mouth, how far apart are the eyes, etc). It's a much simpler problem than the general image detection. Imagine how many different types of dogs you can encounter at various angles, sizes, colors, perspectives. For that it's much more of a black box as your model would have to support detecting a dog from behind, side, front, above, sitting down, walking, jumping, standing, dogs with long tails, short tails, long hair, short hair, etc. Now think about a face. Most faces are very much alike. https://github.com/ageitgey/face_recognition https://github.com/ageitgey/face_recognition

zwkrt 7y ago

This is such an interesting topic and I really don't know where to stand on it. On one hand, this is an argument that systems such as laws, lending practices, etc. should be based on repeatable and explainable rules. Rules will end up with some bias, but the bias can be dissected and argued about until we reach some consensus that the bias is acceptable/desiresble. In the case of lending, maybe we consider credit history a fair input metric but zip-code an unfair metric, for example. On the other hand, systems such as laws and lending, which are inherently about social interaction, do tend to have some unspecified human element. Judges interpret the law and apply it to specific cases, underwriters have latitude to make exceptions under certain vague circumstances, teachers may regrade a paper upon realizing they misspoke in a lecture. This is a feature, not a bug--if our social systems have no room for empathy then there is a big problem. So now that AI is "unexplainable", how is this worse than the unexplainability of human systems? You can ask a human why they took some action, but their explanation tends to be incomplete, wrong, a lie, or maybe they don't recall. I once was given the opportunity to lease a top-floor apartment in my building because I had a daily chat with the receptionist. All fair housing laws were followed, I just happened to be the first to know because I had such frequent interaction. If you were to ask the receptionist why I got the apartment she probably wouldn't say "zwkrt tells me bad jokes and we share pictures of our pets", but that probably is part of the reason. I think fundamentally we understand that humans share a way of life and a set of ineffable values, and we believe that computer-generated results do not share these values. Unfortunately I do not have a conclusion, just a formulation of a problem.

atoav 7y ago
The present systems have more attributes than “they somehow work, but slow”. Think about separation of power and decentralization. Slowness is a feature here. If you want to radically change a society where decisions are carried out by any number of individual actors, you need to take them with you by either using military force, money or convincing them otherwise. In any case this is a project that could also fail. With an algorithm, you could change it today and it would be in action without many even noticing it. I could get behind a simple and transparent tax system, where you just see in realtime what money you give while beeing sure that big company has to do the same, without gaming the system. But I am not sure the system that decides on these rules should be another system.

willvarfar 7y ago

Do you put the same requirement on natural neural nets? Is the decision reached by human members of a jury, for example, explainable in the way you mean?

Mirioron 7y ago

Can you explain how you, as a human being, understand handwriting? Can you explain it in detail without handwaving some stuff away as "now I detect the pattern"? Because I'm pretty sure you can't, but you use it for basically everything in life. So, should a person's eyes not be trusted when it comes to "something important"?

afthonos 7y ago
Given that eyewitness testimony is consistently unreliable, biased, and internally inconsistent, no, they shouldn’t. Human judgement is the best we can do so far, but it is hardly a worthwhile goal, especially with systems that take us from affecting individuals on a person-to-person level to influencing the interactions of millions of people with their government. Algorithms can be explainable, and I agree that anything that affects your standing in the eyes of government should have that as a minimum requirement.
rizzin 7y ago
>So, should a person's eyes not be trusted when it comes to "something important"? A documentary called "Murder on a Sunday Morning": https://www.youtube.com/watch?v=LFLbptkb1eM https://www.youtube.com/watch?v=LFLbptkb1eM A woman was murdered in front of her husband. The man saw the attacker up close and was the only eyewitness. He accused a completely innocent man, and it took half a year of jail and court-related turmoils for this to get cleared up.
honzzz 7y ago
> Can you explain how you, as a human being, understand handwriting? I do not want to speak for the parent but I think you might be on a different level of abstraction when talking about "explaining" things. Consider this for example: someone writes a letter to your boss that you should be fired. That someone does not need to explain the process of writing but probably should be required to explain why you should be fired.
colllectorof 7y ago
>Can you explain how you, as a human being, understand handwriting? We explain how to understand handwriting to every single child that goes to school. It's not the answer you want, but it's the answer that actually matters here. Trying to equivocate AI and human cognition in this way is completely disingenuous. Human reasoning is not 100% reliable, but we know very well in which ways it's unreliable and how to deal with it. We have shared biology and millennia of experience trying to empathize and communicate with others.

sireat 7y ago

The problem is that most machine learning model choices are humanly indecipherable. You can understand how the model works and the math behind it but you will be hard pressed to understand the exact path behind a particular choice. Decision trees are one exception to this. Most humans can understand those.