5 ms·

I really disagree and think the whole "There's no way to gauge results!" meme is low-impact FUD. FUD that isn't particularly dangerous so people that don't know

I really disagree and think the whole "There's no way to gauge results!" meme is low-impact FUD. FUD that isn't particularly dangerous so people that don't know any better just believe it (like python can't be performant because of the GIL or macs are better than PCs or some other inanity). > As exciting as their performance gains have been, though, there’s a troubling fact about modern neural networks: Nobody knows quite how they work. And that means no one can predict when they might fail. Nonsense! Cross validation. Develop hypotheses, develop subsets of data to prove or disprove given hypotheses, observe how the network reacts. All of these people complaining about not being able to understand what's going on are either reporters, bloggers, or machine learning dabblers looking to say something seemingly unconventional. From your linked article, which gives more specifics as to the argument: > Because ML systems are opaque, you cannot really reason about what they do. Yes, it is possible to reason about a system even if it is "opaque"; the discipline is called reverse engineering. Or the scientific method. > Also, you can’t do modular (as in module-by-module) verification. You can do "modular verification" in a variety of ways. Start with analyzing the behavior of each layer and how that changes as you incorporate more layers. It's beyond the scope of this comment to go into it beyond surface level, but there are a lot of papers written about it, google "analyzing neural network hidden activations" or something. > And you can never be sure what they’ll do about a situation never encountered before. Humans can never be sure what they'll do about a situation they haven't encountered. Or engineers. They can simulate the events that they can think of, but we can also do that with a neural network. > Finally, when fixing a bug (e.g. by adding the buggy situation + correct output to the learning set), you can never be sure (without a lot of testing) that the system has fixed “the full bug” and not just some manifestations of it. Fixing the "full bug" is often not something that can be done in traditional software development "without a lot of testing". Machine learning works the same way. If you want if/then statements, use a decision tree. If you want strong accuracy on predictions, use a neural network. It helps to know what you're doing when verifying results. You will run into trouble if you don't know what you're doing, as per common sense.

semi-extrinsic 10y ago
If someone wants an analogy: Nobody understands exactly how painkillers (aspirin, paracetamol etc.) work on the molecular level. Yet they are generally held to be quite useful. Edit: electroconvulsive therapy is an even more extreme example; we have no clue how it works, but it's very effective on severe depression. We only know it works because some Italians back in the olden days, before ethics committees were invented, decided to electro-shock a bunch of "crazy people" to see what happened. The reason they tried it was that electroshocking of pigs for slaughter had been observed to give a temporary anaesthetic effect.
- peterbonney 10y ago
  Or: nobody understands exactly how your doctor's brain reaches its conclusion about what's wrong with you, but you still trust it.
  - cscurmudgeon 10y ago
    It is not like the doctor can explain in English when you ask him for details. Not all.
    - vertex-four 10y ago
      Or, to put it in other words - how your car's AI decides that there's five cars around you probably isn't that interesting. How it decides that based on its input, it should probably slam the brakes, is interesting.
    - deleted 10y ago
      [deleted]
    - Joof 10y ago
      I have a difficult time getting psychiatrists to do exactly this, so no change here in that area.
- cscurmudgeon 10y ago
  So how do we get the first sick volunteers for Dr. Neural Network?
yoav_hollander 10y ago
Well yes, you _can_ verify ML-based systems (and I discuss briefly how in the linked article). It is just that this is much harder with such systems than with HW or SW systems for which "you have the source code". What I meant by "Because ML systems are opaque, you cannot really reason about what they do" (perhaps I was not being clear) was this: You can indeed _observe_ what they do, but being able to actually inspect the source makes verification much easier and more reliable: You know what parts of the logic you have covered, you can think of "danger areas" (and direct testing to them), and you can simply check whether all the cases you can think about have been covered in the source. With opaque systems, you have no idea whether e.g. your ML-based autonomous vehicle will recognize people-painted-on-a-bus as people-on-the-road, until you actually test for that. And then you have to test people-painted-on-a-truck. What I meant regarding "modular verification" is that you can check sub-modules according to some spec (or at least according to informal comments). This is quite different from what you can do when analyzing NN layers. I suspect you are right in claiming (in your last paragraph) that one will always have to choose between "more understandable" and "more accurate". But I think we can do various things (the DARPA suggestion being one of them) to make the "more accurate" solution more verifiable.
Inlinked 10y ago
There is no strong theoretical foundation for very complex and/or deep models. There are ways to gauge results, but right now, the media is right in that there are risks. The pneumonia project referenced in the article had Caruana voting against implementing the most accurate model: A neural net. Instead they went with a way less accurate logistic regression model, one they could safely implement in production, inspect, explain, and defend to the doctors. Nobody does quite know why neural networks work so well. There is a Nobel Prize waiting there for someone or some team to solve this with mathematical (or physics) rigor. Nodes in neural network layers can represent multiple features, or share feature representations. Do we know if a neural net (and which part) is targeting skin color, or acne? Do we know that credit risk models are targeting sex, even though we left out this feature (it may infer this from other features)?. Depending on the application, this is important to know for certain. Reverse engineering sure does work, but can we fully find out the source code from a program, just by fuzzing inputs and looking at outputs? Or are we only looking at (perhaps a small part of) its behavior? > You will run into trouble if you don't know what you're doing Likewise: You will run into trouble, if you don't know for sure what your models are doing. http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tags-black-people-as-gorillas-showing-limits-of-algorithms/ http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag... > as per common sense https://en.wikipedia.org/wiki/Commonsense_reasoning#Commonsense_knowledge_problem https://en.wikipedia.org/wiki/Commonsense_reasoning#Commonse...

semi-extrinsic 10y ago

If someone wants an analogy: Nobody understands exactly how painkillers (aspirin, paracetamol etc.) work on the molecular level. Yet they are generally held to be quite useful. Edit: electroconvulsive therapy is an even more extreme example; we have no clue how it works, but it's very effective on severe depression. We only know it works because some Italians back in the olden days, before ethics committees were invented, decided to electro-shock a bunch of "crazy people" to see what happened. The reason they tried it was that electroshocking of pigs for slaughter had been observed to give a temporary anaesthetic effect.

peterbonney 10y ago
Or: nobody understands exactly how your doctor's brain reaches its conclusion about what's wrong with you, but you still trust it.
- cscurmudgeon 10y ago
  It is not like the doctor can explain in English when you ask him for details. Not all.
  - vertex-four 10y ago
    Or, to put it in other words - how your car's AI decides that there's five cars around you probably isn't that interesting. How it decides that based on its input, it should probably slam the brakes, is interesting.
  - deleted 10y ago
    [deleted]
  - Joof 10y ago
    I have a difficult time getting psychiatrists to do exactly this, so no change here in that area.
cscurmudgeon 10y ago
So how do we get the first sick volunteers for Dr. Neural Network?

yoav_hollander 10y ago

Well yes, you _can_ verify ML-based systems (and I discuss briefly how in the linked article). It is just that this is much harder with such systems than with HW or SW systems for which "you have the source code". What I meant by "Because ML systems are opaque, you cannot really reason about what they do" (perhaps I was not being clear) was this: You can indeed _observe_ what they do, but being able to actually inspect the source makes verification much easier and more reliable: You know what parts of the logic you have covered, you can think of "danger areas" (and direct testing to them), and you can simply check whether all the cases you can think about have been covered in the source. With opaque systems, you have no idea whether e.g. your ML-based autonomous vehicle will recognize people-painted-on-a-bus as people-on-the-road, until you actually test for that. And then you have to test people-painted-on-a-truck. What I meant regarding "modular verification" is that you can check sub-modules according to some spec (or at least according to informal comments). This is quite different from what you can do when analyzing NN layers. I suspect you are right in claiming (in your last paragraph) that one will always have to choose between "more understandable" and "more accurate". But I think we can do various things (the DARPA suggestion being one of them) to make the "more accurate" solution more verifiable.

Inlinked 10y ago

There is no strong theoretical foundation for very complex and/or deep models. There are ways to gauge results, but right now, the media is right in that there are risks. The pneumonia project referenced in the article had Caruana voting against implementing the most accurate model: A neural net. Instead they went with a way less accurate logistic regression model, one they could safely implement in production, inspect, explain, and defend to the doctors. Nobody does quite know why neural networks work so well. There is a Nobel Prize waiting there for someone or some team to solve this with mathematical (or physics) rigor. Nodes in neural network layers can represent multiple features, or share feature representations. Do we know if a neural net (and which part) is targeting skin color, or acne? Do we know that credit risk models are targeting sex, even though we left out this feature (it may infer this from other features)?. Depending on the application, this is important to know for certain. Reverse engineering sure does work, but can we fully find out the source code from a program, just by fuzzing inputs and looking at outputs? Or are we only looking at (perhaps a small part of) its behavior? > You will run into trouble if you don't know what you're doing Likewise: You will run into trouble, if you don't know for sure what your models are doing. http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tags-black-people-as-gorillas-showing-limits-of-algorithms/ http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag... > as per common sense https://en.wikipedia.org/wiki/Commonsense_reasoning#Commonsense_knowledge_problem https://en.wikipedia.org/wiki/Commonsense_reasoning#Commonse...