4 ms·

Yes, they are highlighting the importance of research towards model explainability.

by NieDzejkob 5y ago

slver 5y ago
I believe that’s impossible.
- TaylorAlexander 5y ago
  ML explainability is a wide field with a lot of success. For example you can discover what features are activating which detections. You comment, taken literally, suggests that research in this field is impossible. That is not the case.
  - sigstoat 5y ago
    when the activated feature in an image recognition net looks like a lovecraftian horror, that doesn’t explain how the net came up with “turtle”. explainability is going to have a rough time for the same reason ai alignment is going to have a rough time. people think they can explain decisions (technical and moral) far more effectively than they actually can.
    - TaylorAlexander 5y ago
      Yes it is a difficult field that will not solve every problem with ML, but I wouldn’t say that performing research in the field is impossible.
      - slver 5y ago
        I judge by our general inability to explain most of our intuitive decisions (intuition is basically trained pattern recognition neural network without reflection, much like artificial ML). A great sports player often makes a lousy coach. They often can't articulate how they play, how they move, and how they think, except in the broadest strokes. All our formal/reflective models are evolved entirely independently of our intuition. They're supported by our intuition (release ball, ball drops, gravity) as a heuristic, but not explained by it. They're independent in terms of logical frame. I suspect a similar thing will occur in ML. We'll have our black box ML that produces great results which we can't explain. And we'll have other NN models that arrive at reflective models, which however are far less insightful on their own (at least per watt consumed, let's say). The trouble is that we expect to sit down and debug a series of equations in a NN and come up with a "factual nugget" or a few, that explains what happened. Actually correlation often doesn't work this way. You just correlate dozens of mundane factors that vary by one degree between outcomes, and you happen to be able to produce a solid result from that. Expecting specific models in a NN is a bit like inspecting a dog photo on your phone under a magnifying glass, expecting to learn more about the dog. Instead you cease seeing a dog, and start seeing arbitrary colored pixels.
      - IQunder130 5y ago
        The problem is with your assumption that the decision function can be reduced to a nice, clean, closed form that a human's brain can consciously conceptualize. It might as well be a tangled mess of a thousand parameters you can't reduce any further without losing predictive power. There is no particular reason why anything should be simple.

Yes, they are highlighting the importance of research towards model explainability.

slver 5y ago

I believe that’s impossible.

TaylorAlexander 5y ago
ML explainability is a wide field with a lot of success. For example you can discover what features are activating which detections. You comment, taken literally, suggests that research in this field is impossible. That is not the case.
- sigstoat 5y ago
  when the activated feature in an image recognition net looks like a lovecraftian horror, that doesn’t explain how the net came up with “turtle”. explainability is going to have a rough time for the same reason ai alignment is going to have a rough time. people think they can explain decisions (technical and moral) far more effectively than they actually can.
  - TaylorAlexander 5y ago
    Yes it is a difficult field that will not solve every problem with ML, but I wouldn’t say that performing research in the field is impossible.
    - slver 5y ago
      I judge by our general inability to explain most of our intuitive decisions (intuition is basically trained pattern recognition neural network without reflection, much like artificial ML). A great sports player often makes a lousy coach. They often can't articulate how they play, how they move, and how they think, except in the broadest strokes. All our formal/reflective models are evolved entirely independently of our intuition. They're supported by our intuition (release ball, ball drops, gravity) as a heuristic, but not explained by it. They're independent in terms of logical frame. I suspect a similar thing will occur in ML. We'll have our black box ML that produces great results which we can't explain. And we'll have other NN models that arrive at reflective models, which however are far less insightful on their own (at least per watt consumed, let's say). The trouble is that we expect to sit down and debug a series of equations in a NN and come up with a "factual nugget" or a few, that explains what happened. Actually correlation often doesn't work this way. You just correlate dozens of mundane factors that vary by one degree between outcomes, and you happen to be able to produce a solid result from that. Expecting specific models in a NN is a bit like inspecting a dog photo on your phone under a magnifying glass, expecting to learn more about the dog. Instead you cease seeing a dog, and start seeing arbitrary colored pixels.
    - IQunder130 5y ago
      The problem is with your assumption that the decision function can be reduced to a nice, clean, closed form that a human's brain can consciously conceptualize. It might as well be a tangled mess of a thousand parameters you can't reduce any further without losing predictive power. There is no particular reason why anything should be simple.