5 ms·

It's not that it is a black box that can somehow be unblacked. Neural networks are inherently messy things that have contrived, complex and partial or ad-hoc re

It's not that it is a black box that can somehow be unblacked. Neural networks are inherently messy things that have contrived, complex and partial or ad-hoc representations

burtonator 4y ago
Also, If you look at each layer as just a vector, how the heck do you describe that so that it's easy to understand? I think this might be perpetually difficult to diagnose. Maybe there could be a tool that could show WHY a decision was made but not sure that you could identify bias in deep networks before hand.
- ewuhic 4y ago
  Isn't the "why" the weights of the model themselves?
  - sdenton4 4y ago
    People want brief explanations of reasons for outputs, and a 10gb pile of weights isn't really what they mean. Human explanations meanwhile are often invented to fit evidence to personal biases and beliefs, and are thus typically deeply flawed. But we're more ok with humans making suspect decisions than ML, in many cases.
version_five 4y ago
Yeah, when people want to see e.g. an image classification model explain the different features it saw in the image and the weights it assigned them in making it's decision (this is an example in the article), they are asking for something that isn't what the model does. ML models have tacit knowledge in a sense, you can't tractably write down a process for it. That's not to say you can't describe the situations in which a model works.
drdeca 4y ago
Are you familiar with the "circuits" thread on distill.pub ? ( https://distill.pub/2020/circuits/ https://distill.pub/2020/circuits/ ) Messy and complex, yes, but not altogether immune to analysis. And, if the training data is diverse enough, it appears that the individual neurons can reflect things we find meaningful, while being expressed in terms of neurons in previous layers which we also find semantically meaningful, in a way we can find comprehensible. Of course, the amount of time and effort needed to collectively understand the entirety of such a network (to the point that a similar network could be made by people choosing weights by hand (not copying the specific numbers, only the interpreted meanings/reasons of the numbers), and producing something which is not too much worse than the trained network, would be gargantuan, and I suspect it might require multiple generations, possibly even many. But, I don't think it is impossible? (presumably it will never happen, because it would not come anywhere close to being worth it to do the whole thing, but, still.)
- mr_toad 4y ago
  > the individual neurons can reflect things we find meaningful, while being expressed in terms of neurons in previous layers It’s easy to find relationships & explanations of how a network seems to work that seem to hold - until they don’t. Consider a circle. Assume that the y-coordinate corresponds to the centre of the circle. You can now describe which points lie inside the circle as a function of the x-coordinate. You can vary y to a certain extent, which changes the function, but not fundamentally. But now shift the y-coordinate outside the circle. Now your function of x, which seemed so convincing is entirely useless. How does the probability of being inside the circle depend on x? What weight does the model ascribe ascribed to x in making the decision? The answer always has to be ‘it depends’. And that’s just a model with two parameters.
  - drdeca 4y ago
    Sorry, I don't know what you mean by the thing about "the y-coordinate corresponds to the centre of the circle" . Do you mean that the y coordinate is the y coordinate of the center of the circle? or, is the y coordinate a position and not just a number? Do you just mean that you have one parameter which determines the center of the circle, and another (which you called x), which determines the radius? no, that can't be it, because you then talk about moving the y coordinate outside the circle. Do you mean, if you have a given y value, and you are describing "is the point (x,y) inside the circle" as a function of x, then if you pick the y value to be further than r away from the the y coordinate of the center of the circle, where r is the radius, that the function stops depending on x and for all x, is "no" ? I'm really not sure what point you are trying to make. I'm not convinced that whatever your point is applies to the results I linked. iirc, they did, for a handful of the neurons, try replacing them with human-designed ones using the same principles, and I believe that that generally worked without all that much degradation of the functioning? I could be accidentally making that up though (as in, possibly I'm wrong in thinking they did that)

It's not that it is a black box that can somehow be unblacked. Neural networks are inherently messy things that have contrived, complex and partial or ad-hoc representations

burtonator 4y ago

Also, If you look at each layer as just a vector, how the heck do you describe that so that it's easy to understand? I think this might be perpetually difficult to diagnose. Maybe there could be a tool that could show WHY a decision was made but not sure that you could identify bias in deep networks before hand.

ewuhic 4y ago
Isn't the "why" the weights of the model themselves?
- sdenton4 4y ago
  People want brief explanations of reasons for outputs, and a 10gb pile of weights isn't really what they mean. Human explanations meanwhile are often invented to fit evidence to personal biases and beliefs, and are thus typically deeply flawed. But we're more ok with humans making suspect decisions than ML, in many cases.

version_five 4y ago

Yeah, when people want to see e.g. an image classification model explain the different features it saw in the image and the weights it assigned them in making it's decision (this is an example in the article), they are asking for something that isn't what the model does. ML models have tacit knowledge in a sense, you can't tractably write down a process for it. That's not to say you can't describe the situations in which a model works.

drdeca 4y ago

Are you familiar with the "circuits" thread on distill.pub ? ( https://distill.pub/2020/circuits/ https://distill.pub/2020/circuits/ ) Messy and complex, yes, but not altogether immune to analysis. And, if the training data is diverse enough, it appears that the individual neurons can reflect things we find meaningful, while being expressed in terms of neurons in previous layers which we also find semantically meaningful, in a way we can find comprehensible. Of course, the amount of time and effort needed to collectively understand the entirety of such a network (to the point that a similar network could be made by people choosing weights by hand (not copying the specific numbers, only the interpreted meanings/reasons of the numbers), and producing something which is not too much worse than the trained network, would be gargantuan, and I suspect it might require multiple generations, possibly even many. But, I don't think it is impossible? (presumably it will never happen, because it would not come anywhere close to being worth it to do the whole thing, but, still.)

mr_toad 4y ago
> the individual neurons can reflect things we find meaningful, while being expressed in terms of neurons in previous layers It’s easy to find relationships & explanations of how a network seems to work that seem to hold - until they don’t. Consider a circle. Assume that the y-coordinate corresponds to the centre of the circle. You can now describe which points lie inside the circle as a function of the x-coordinate. You can vary y to a certain extent, which changes the function, but not fundamentally. But now shift the y-coordinate outside the circle. Now your function of x, which seemed so convincing is entirely useless. How does the probability of being inside the circle depend on x? What weight does the model ascribe ascribed to x in making the decision? The answer always has to be ‘it depends’. And that’s just a model with two parameters.
- drdeca 4y ago
  Sorry, I don't know what you mean by the thing about "the y-coordinate corresponds to the centre of the circle" . Do you mean that the y coordinate is the y coordinate of the center of the circle? or, is the y coordinate a position and not just a number? Do you just mean that you have one parameter which determines the center of the circle, and another (which you called x), which determines the radius? no, that can't be it, because you then talk about moving the y coordinate outside the circle. Do you mean, if you have a given y value, and you are describing "is the point (x,y) inside the circle" as a function of x, then if you pick the y value to be further than r away from the the y coordinate of the center of the circle, where r is the radius, that the function stops depending on x and for all x, is "no" ? I'm really not sure what point you are trying to make. I'm not convinced that whatever your point is applies to the results I linked. iirc, they did, for a handful of the neurons, try replacing them with human-designed ones using the same principles, and I believe that that generally worked without all that much degradation of the functioning? I could be accidentally making that up though (as in, possibly I'm wrong in thinking they did that)