4 ms·

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. Seeing which parts of a mod

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. Seeing which parts of a model (they aren't neurons) light up when shown a face doesn't necessarily indicate understanding. The model is a complex web of numbers representing a massively compressed data space. It could easily be that what you see light up when shown a face only indicates what specific part of the model is housing the compresses data related to recognizing specific facial features.

the_af 1y ago
I think this could be seen as a proxy for evidence that there's some degree of reasoning, if we think we can identify specialized features that always become involved in some kinds of outputs. It's not proof, but it's not nothing either. It has some parallels about research on human brains are conducted, right?
- _heimdall 1y ago
  It does have parallels to the human brain, absolutely. We've been studying the human brain in similar ways for much longer though and we still don't know much about it. We do know what areas of a human brain often light up in response to various conditions. We don't know why that is though, or how it actually works. Maybe more importantly for LLMs, we don't know how human memory works, where it is stored, how to recognize or even define consciousness, etc. Seeing what areas of a brain or an LLM light up can be interesting, but I'd be very cautious trying to read much into it.
neonspark 1y ago
> Seeing which parts of a model (they aren't neurons)… I thought models were composed of neural network layers, among other things. Are these data structures called something different?
- _heimdall 1y ago
  That point may not have been relevant for me to include. I was getting at the idea that a neuron is a very specific feature of a biological brain, regardless of what AI researchers may call their hardware they aren't made of neurons.
  - jychang 1y ago
    1. They are neurons, whether you like it or not. A binary tree may not have squirrels living in them, but it's still a tree, even though the word "tree" here is defined differently than from biology. Or are you going to say a binary tree is not a tree? 2. You are about 5 years behind in terms of the research. Look into hierarchical feature representation and how MLP neurons work. (Or even in older CNNs and RNNs etc). And I'm willingly using the word "neuron" instead of "feature" here because while I know "feature" is more correct in general, there are definitely small toy models where you can pinpoint an individual neuron to represent a feature such as a face.
    - _heimdall 1y ago
      What were you getting at with the MLP example? MLPs do a great job with perception abilities and I get that they use the term neuron frequently. I disagree with the use of the name there that's all, similarly I disagree that LLMs are AI but here we are. Using the term neuron there and meaning it literally is like calling an airplane a bird. I get that the colloquial use exists, but no one thinks they are literal birds.
      - jychang 1y ago
        Do you also disagree with the use of the name “tree” in a computer science class? Again, nobody thinks trees in computer science contains squirrels, nobody thinks airplanes are birds, and nobody thinks a neuron in a ML model contains axons and dendrites. This is a weird hill to die on. Are you gonna complain that the word “photograph” is “light writing” but in reality nobody is writing anything so therefore the word is wrong?
        1y ago

the_af 1y ago

I think this could be seen as a proxy for evidence that there's some degree of reasoning, if we think we can identify specialized features that always become involved in some kinds of outputs. It's not proof, but it's not nothing either. It has some parallels about research on human brains are conducted, right?

_heimdall 1y ago
It does have parallels to the human brain, absolutely. We've been studying the human brain in similar ways for much longer though and we still don't know much about it. We do know what areas of a human brain often light up in response to various conditions. We don't know why that is though, or how it actually works. Maybe more importantly for LLMs, we don't know how human memory works, where it is stored, how to recognize or even define consciousness, etc. Seeing what areas of a brain or an LLM light up can be interesting, but I'd be very cautious trying to read much into it.

neonspark 1y ago

> Seeing which parts of a model (they aren't neurons)… I thought models were composed of neural network layers, among other things. Are these data structures called something different?

_heimdall 1y ago
That point may not have been relevant for me to include. I was getting at the idea that a neuron is a very specific feature of a biological brain, regardless of what AI researchers may call their hardware they aren't made of neurons.
- jychang 1y ago
  1. They are neurons, whether you like it or not. A binary tree may not have squirrels living in them, but it's still a tree, even though the word "tree" here is defined differently than from biology. Or are you going to say a binary tree is not a tree? 2. You are about 5 years behind in terms of the research. Look into hierarchical feature representation and how MLP neurons work. (Or even in older CNNs and RNNs etc). And I'm willingly using the word "neuron" instead of "feature" here because while I know "feature" is more correct in general, there are definitely small toy models where you can pinpoint an individual neuron to represent a feature such as a face.
  - _heimdall 1y ago
    What were you getting at with the MLP example? MLPs do a great job with perception abilities and I get that they use the term neuron frequently. I disagree with the use of the name there that's all, similarly I disagree that LLMs are AI but here we are. Using the term neuron there and meaning it literally is like calling an airplane a bird. I get that the colloquial use exists, but no one thinks they are literal birds.
    - jychang 1y ago
      Do you also disagree with the use of the name “tree” in a computer science class? Again, nobody thinks trees in computer science contains squirrels, nobody thinks airplanes are birds, and nobody thinks a neuron in a ML model contains axons and dendrites. This is a weird hill to die on. Are you gonna complain that the word “photograph” is “light writing” but in reality nobody is writing anything so therefore the word is wrong?
      - 1y ago