6 ms·

>Clinicians are currently unaware of distinct retinal feature variations between males and females, highlighting the importance of model explainability for this

>Clinicians are currently unaware of distinct retinal feature variations between males and females, highlighting the importance of model explainability for this task. If I'm reading this correctly what they're saying is that since we don't currently know the difference between male and female retinas, being able to explain what the ML black box is doing is important. But from what I can see in the paper they basically don't know what the black box is doing, they really don't understand what features their tool has isolated. I might be misunderstanding though?

oogabooga123 5y ago
Your question is confusing, it might be that you are using “feature” in an ML sense and the quote refers to human describable distinctions we know about? But I still don’t know how to parse your question. The model can predict male vs female retinas but they don’t understand why. What exactly are you asking?
- TaylorAlexander 5y ago
  ML models have layers, and neurons in a given layer detect “features” in the image or in the previous layer. So yes I believe the person meant which structures in the image are activating the network. Which is a well studied area so it is surprising the authors didn’t explore that.
NieDzejkob 5y ago
Yes, they are highlighting the importance of research towards model explainability.
- slver 5y ago
  I believe that’s impossible.
  - TaylorAlexander 5y ago
    ML explainability is a wide field with a lot of success. For example you can discover what features are activating which detections. You comment, taken literally, suggests that research in this field is impossible. That is not the case.
    - sigstoat 5y ago
      when the activated feature in an image recognition net looks like a lovecraftian horror, that doesn’t explain how the net came up with “turtle”. explainability is going to have a rough time for the same reason ai alignment is going to have a rough time. people think they can explain decisions (technical and moral) far more effectively than they actually can.
      - TaylorAlexander 5y ago
        Yes it is a difficult field that will not solve every problem with ML, but I wouldn’t say that performing research in the field is impossible.
        slver 5y ago
        I judge by our general inability to explain most of our intuitive decisions (intuition is basically trained pattern recognition neural network without reflection, much like artificial ML). A great sports player often makes a lousy coach. They often can't articulate how they play, how they move, and how they think, except in the broadest strokes. All our formal/reflective models are evolved entirely independently of our intuition. They're supported by our intuition (release ball, ball drops, gravity) as a heuristic, but not explained by it. They're independent in terms of logical frame. I suspect a similar thing will occur in ML. We'll have our black box ML that produces great results which we can't explain. And we'll have other NN models that arrive at reflective models, which however are far less insightful on their own (at least per watt consumed, let's say). The trouble is that we expect to sit down and debug a series of equations in a NN and come up with a "factual nugget" or a few, that explains what happened. Actually correlation often doesn't work this way. You just correlate dozens of mundane factors that vary by one degree between outcomes, and you happen to be able to produce a solid result from that. Expecting specific models in a NN is a bit like inspecting a dog photo on your phone under a magnifying glass, expecting to learn more about the dog. Instead you cease seeing a dog, and start seeing arbitrary colored pixels.
hervature 5y ago
We're not at the point of explaining complicated models with a straight face. Based on the saliency maps, it looks like the model has learned something around the bright circles (or is it the blind spot? Not an ophthalmologist). Makes me think the network can reverse engineer distortions in the light to get curvature of the lens which might be indicative of gender differences.
- drdeca 5y ago
  The circuits thread at distil.pub seems promising to me (though I’m not a ML researcher, so others would be better able to judge this) That’s not to say it is quite to the point of completely explaining each classification, but, I do think it is very promising.
thaumasiotes 5y ago
I don't really understand your confusion? They say the following: - This model can distinguish photographs of a male retina from photographs of a female retina. - We don't know, ourselves, how to do that. - We would like to be able to determine, from looking at the model, what features it's using to draw the distinction. What's weird?
- Traster 5y ago
  What I'm confused by is that they say this is important to do, but then don't actually seem to do it?
  - joe_the_user 5y ago
    They don't give an explanation because they don't know how to give an explanation - many if not most ML model lack an easy explanation presently, they just spit out answers. They are saying "someone should do this because it's important even though we don't (presently) know how to do this".
    - aaron-santos 5y ago
      I'm open to learning why class activation maps (CAM) would or wouldn't be a good place to start.
    - thegginthesky 5y ago
      Wouldn't model explainers, such as Shap[0] or LIME[1], be able to assist in explaining these results? There are some very interesting advances in the space of model explainability that merit at least a try. [0]https://github.com/slundberg/shap https://github.com/slundberg/shap [1]https://github.com/marcotcr/lime https://github.com/marcotcr/lime
      - Mehdi2277 5y ago
        Have you ever used them? They're pretty but not things I put much trust in and the other awkward issue is there's been a variety of papers that do an analysis of many interpretability algorithms and find some very weak properties. One example is train a model M, run something like SHAP, delete the features marked important in the original dataset, train a new model M2 and see minimal performance difference. At best that tells us many models have large number of ways to get there answer and the interpretability algorithm is giving you one. If your goal was understanding what distinguishes the data and not just specific model you failed. At worst it tells you the local way most interpretability algorithms work is just not revealing. The local nature (most are heavily gradient based) when working with data like images where individual pixels mean little also puts a lot of skepticism for me. There are a decent number of interpretability algorithms that look like they're just running edge detection which makes sense with gradient part but is poor for interpretability.

oogabooga123 5y ago

Your question is confusing, it might be that you are using “feature” in an ML sense and the quote refers to human describable distinctions we know about? But I still don’t know how to parse your question. The model can predict male vs female retinas but they don’t understand why. What exactly are you asking?

TaylorAlexander 5y ago
ML models have layers, and neurons in a given layer detect “features” in the image or in the previous layer. So yes I believe the person meant which structures in the image are activating the network. Which is a well studied area so it is surprising the authors didn’t explore that.

NieDzejkob 5y ago

Yes, they are highlighting the importance of research towards model explainability.

slver 5y ago
I believe that’s impossible.
- TaylorAlexander 5y ago
  ML explainability is a wide field with a lot of success. For example you can discover what features are activating which detections. You comment, taken literally, suggests that research in this field is impossible. That is not the case.
  - sigstoat 5y ago
    when the activated feature in an image recognition net looks like a lovecraftian horror, that doesn’t explain how the net came up with “turtle”. explainability is going to have a rough time for the same reason ai alignment is going to have a rough time. people think they can explain decisions (technical and moral) far more effectively than they actually can.
    - TaylorAlexander 5y ago
      Yes it is a difficult field that will not solve every problem with ML, but I wouldn’t say that performing research in the field is impossible.
      - slver 5y ago
        I judge by our general inability to explain most of our intuitive decisions (intuition is basically trained pattern recognition neural network without reflection, much like artificial ML). A great sports player often makes a lousy coach. They often can't articulate how they play, how they move, and how they think, except in the broadest strokes. All our formal/reflective models are evolved entirely independently of our intuition. They're supported by our intuition (release ball, ball drops, gravity) as a heuristic, but not explained by it. They're independent in terms of logical frame. I suspect a similar thing will occur in ML. We'll have our black box ML that produces great results which we can't explain. And we'll have other NN models that arrive at reflective models, which however are far less insightful on their own (at least per watt consumed, let's say). The trouble is that we expect to sit down and debug a series of equations in a NN and come up with a "factual nugget" or a few, that explains what happened. Actually correlation often doesn't work this way. You just correlate dozens of mundane factors that vary by one degree between outcomes, and you happen to be able to produce a solid result from that. Expecting specific models in a NN is a bit like inspecting a dog photo on your phone under a magnifying glass, expecting to learn more about the dog. Instead you cease seeing a dog, and start seeing arbitrary colored pixels.

hervature 5y ago

We're not at the point of explaining complicated models with a straight face. Based on the saliency maps, it looks like the model has learned something around the bright circles (or is it the blind spot? Not an ophthalmologist). Makes me think the network can reverse engineer distortions in the light to get curvature of the lens which might be indicative of gender differences.

drdeca 5y ago
The circuits thread at distil.pub seems promising to me (though I’m not a ML researcher, so others would be better able to judge this) That’s not to say it is quite to the point of completely explaining each classification, but, I do think it is very promising.

thaumasiotes 5y ago

I don't really understand your confusion? They say the following: - This model can distinguish photographs of a male retina from photographs of a female retina. - We don't know, ourselves, how to do that. - We would like to be able to determine, from looking at the model, what features it's using to draw the distinction. What's weird?

Traster 5y ago
What I'm confused by is that they say this is important to do, but then don't actually seem to do it?
- joe_the_user 5y ago
  They don't give an explanation because they don't know how to give an explanation - many if not most ML model lack an easy explanation presently, they just spit out answers. They are saying "someone should do this because it's important even though we don't (presently) know how to do this".
  - aaron-santos 5y ago
    I'm open to learning why class activation maps (CAM) would or wouldn't be a good place to start.
  - thegginthesky 5y ago
    Wouldn't model explainers, such as Shap[0] or LIME[1], be able to assist in explaining these results? There are some very interesting advances in the space of model explainability that merit at least a try. [0]https://github.com/slundberg/shap https://github.com/slundberg/shap [1]https://github.com/marcotcr/lime https://github.com/marcotcr/lime
    - Mehdi2277 5y ago
      Have you ever used them? They're pretty but not things I put much trust in and the other awkward issue is there's been a variety of papers that do an analysis of many interpretability algorithms and find some very weak properties. One example is train a model M, run something like SHAP, delete the features marked important in the original dataset, train a new model M2 and see minimal performance difference. At best that tells us many models have large number of ways to get there answer and the interpretability algorithm is giving you one. If your goal was understanding what distinguishes the data and not just specific model you failed. At worst it tells you the local way most interpretability algorithms work is just not revealing. The local nature (most are heavily gradient based) when working with data like images where individual pixels mean little also puts a lot of skepticism for me. There are a decent number of interpretability algorithms that look like they're just running edge detection which makes sense with gradient part but is poor for interpretability.