4 ms·

I think this issue of "Explainable Machine Learning" and interpretability is just going to get more and more important as ML grows. It will also be important fo

I think this issue of "Explainable Machine Learning" and interpretability is just going to get more and more important as ML grows. It will also be important for verifying ML-based systems - another problem area. See [1] for a discussion of both. [1] https://blog.foretellix.com/2016/08/31/machine-learning-verification-and-explainable-ai/ https://blog.foretellix.com/2016/08/31/machine-learning-veri...

dave_sullivan 10y ago
I really disagree and think the whole "There's no way to gauge results!" meme is low-impact FUD. FUD that isn't particularly dangerous so people that don't know any better just believe it (like python can't be performant because of the GIL or macs are better than PCs or some other inanity). > As exciting as their performance gains have been, though, there’s a troubling fact about modern neural networks: Nobody knows quite how they work. And that means no one can predict when they might fail. Nonsense! Cross validation. Develop hypotheses, develop subsets of data to prove or disprove given hypotheses, observe how the network reacts. All of these people complaining about not being able to understand what's going on are either reporters, bloggers, or machine learning dabblers looking to say something seemingly unconventional. From your linked article, which gives more specifics as to the argument: > Because ML systems are opaque, you cannot really reason about what they do. Yes, it is possible to reason about a system even if it is "opaque"; the discipline is called reverse engineering. Or the scientific method. > Also, you can’t do modular (as in module-by-module) verification. You can do "modular verification" in a variety of ways. Start with analyzing the behavior of each layer and how that changes as you incorporate more layers. It's beyond the scope of this comment to go into it beyond surface level, but there are a lot of papers written about it, google "analyzing neural network hidden activations" or something. > And you can never be sure what they’ll do about a situation never encountered before. Humans can never be sure what they'll do about a situation they haven't encountered. Or engineers. They can simulate the events that they can think of, but we can also do that with a neural network. > Finally, when fixing a bug (e.g. by adding the buggy situation + correct output to the learning set), you can never be sure (without a lot of testing) that the system has fixed “the full bug” and not just some manifestations of it. Fixing the "full bug" is often not something that can be done in traditional software development "without a lot of testing". Machine learning works the same way. If you want if/then statements, use a decision tree. If you want strong accuracy on predictions, use a neural network. It helps to know what you're doing when verifying results. You will run into trouble if you don't know what you're doing, as per common sense.
- semi-extrinsic 10y ago
  If someone wants an analogy: Nobody understands exactly how painkillers (aspirin, paracetamol etc.) work on the molecular level. Yet they are generally held to be quite useful. Edit: electroconvulsive therapy is an even more extreme example; we have no clue how it works, but it's very effective on severe depression. We only know it works because some Italians back in the olden days, before ethics committees were invented, decided to electro-shock a bunch of "crazy people" to see what happened. The reason they tried it was that electroshocking of pigs for slaughter had been observed to give a temporary anaesthetic effect.
  - peterbonney 10y ago
    Or: nobody understands exactly how your doctor's brain reaches its conclusion about what's wrong with you, but you still trust it.
    - cscurmudgeon 10y ago
      It is not like the doctor can explain in English when you ask him for details. Not all.
      - vertex-four 10y ago
        Or, to put it in other words - how your car's AI decides that there's five cars around you probably isn't that interesting. How it decides that based on its input, it should probably slam the brakes, is interesting.
      - deleted 10y ago
        [deleted]
      - Joof 10y ago
        I have a difficult time getting psychiatrists to do exactly this, so no change here in that area.
  - cscurmudgeon 10y ago

dave_sullivan 10y ago

I really disagree and think the whole "There's no way to gauge results!" meme is low-impact FUD. FUD that isn't particularly dangerous so people that don't know any better just believe it (like python can't be performant because of the GIL or macs are better than PCs or some other inanity). > As exciting as their performance gains have been, though, there’s a troubling fact about modern neural networks: Nobody knows quite how they work. And that means no one can predict when they might fail. Nonsense! Cross validation. Develop hypotheses, develop subsets of data to prove or disprove given hypotheses, observe how the network reacts. All of these people complaining about not being able to understand what's going on are either reporters, bloggers, or machine learning dabblers looking to say something seemingly unconventional. From your linked article, which gives more specifics as to the argument: > Because ML systems are opaque, you cannot really reason about what they do. Yes, it is possible to reason about a system even if it is "opaque"; the discipline is called reverse engineering. Or the scientific method. > Also, you can’t do modular (as in module-by-module) verification. You can do "modular verification" in a variety of ways. Start with analyzing the behavior of each layer and how that changes as you incorporate more layers. It's beyond the scope of this comment to go into it beyond surface level, but there are a lot of papers written about it, google "analyzing neural network hidden activations" or something. > And you can never be sure what they’ll do about a situation never encountered before. Humans can never be sure what they'll do about a situation they haven't encountered. Or engineers. They can simulate the events that they can think of, but we can also do that with a neural network. > Finally, when fixing a bug (e.g. by adding the buggy situation + correct output to the learning set), you can never be sure (without a lot of testing) that the system has fixed “the full bug” and not just some manifestations of it. Fixing the "full bug" is often not something that can be done in traditional software development "without a lot of testing". Machine learning works the same way. If you want if/then statements, use a decision tree. If you want strong accuracy on predictions, use a neural network. It helps to know what you're doing when verifying results. You will run into trouble if you don't know what you're doing, as per common sense.

semi-extrinsic 10y ago
If someone wants an analogy: Nobody understands exactly how painkillers (aspirin, paracetamol etc.) work on the molecular level. Yet they are generally held to be quite useful. Edit: electroconvulsive therapy is an even more extreme example; we have no clue how it works, but it's very effective on severe depression. We only know it works because some Italians back in the olden days, before ethics committees were invented, decided to electro-shock a bunch of "crazy people" to see what happened. The reason they tried it was that electroshocking of pigs for slaughter had been observed to give a temporary anaesthetic effect.
- peterbonney 10y ago
  Or: nobody understands exactly how your doctor's brain reaches its conclusion about what's wrong with you, but you still trust it.
  - cscurmudgeon 10y ago
    It is not like the doctor can explain in English when you ask him for details. Not all.
    - vertex-four 10y ago
      Or, to put it in other words - how your car's AI decides that there's five cars around you probably isn't that interesting. How it decides that based on its input, it should probably slam the brakes, is interesting.
    - deleted 10y ago
      [deleted]
    - Joof 10y ago
      I have a difficult time getting psychiatrists to do exactly this, so no change here in that area.
- cscurmudgeon 10y ago