4 ms·

I’m not sure it follows that the entire circuit can be subject to this kind of interrogation resulting in a neat story. We can always try to find narratives, bu

I’m not sure it follows that the entire circuit can be subject to this kind of interrogation resulting in a neat story. We can always try to find narratives, but at the end of the day there’s no reason for all the parameter values learned to adhere to some nice human-parsable reason for existing. Edit: Maybe in theory you could “understand” anything arbitrarily large if you just had enough memory to handle very complicated narratives. But I’m doubting that models can always be compressed into a simpler narrative representation.

drdeca 4y ago
It is true that many of the neurons don’t seem appropriate (e.g. mixing two largely unrelated concepts), but, well, that particular issue isn’t a problem with the hypothetical I describe, because “this neuron represents ‘either a fire truck or a nebula’” is still something we can describe. But, yes, I suppose it is possible that there could be important parts of how the network works that aren’t amenable to understanding what a local part is doing in order to achieve a task. Perhaps details in how parts behave which seem meaningless individually, and only have a useful effect when many such parts are put together across many layers. But, I still think that it is important to note that, when trained in a diverse way, representative of how the actual distribution, there are at least, many parts of the functioning which can be understood. Trained NNs aren’t inherently completely inscrutable in how they accomplish things. But yes, you do have a point that we probably can’t really rule out that there are important aspects of the functioning of these networks that can’t be understood locally in a human-meaningful way, even if many parts of it can be understood.