5 ms·

Identifying the dimensions used by the primate brain to decode faces

197 points by pk2200 9y ago

pk2200 9y ago
Here's the paper: http://www.cell.com/cell/pdf/S0092-8674(17)30538-X.pdf http://www.cell.com/cell/pdf/S0092-8674(17)30538-X.pdf
- paulfrancisco 9y ago
  All the faces seem computer generated. It would have been nice if they had used celebrity faces we can all recognize and see what their system comes up with.
  - iandanforth 9y ago
    They are real faces that are morphed along one of the 50 axes they came up with. They look less real because it's standard practice to exclude hair and shoulders and backgrounds from face databases (even if this is totally unrealistic it helps isolated the system under study)
  - rangibaby 9y ago
    You can see some of the faces your brain comes up with here: https://youtu.be/VT9i99D_9gI https://youtu.be/VT9i99D_9gI
gech 9y ago
Can this model be translated into computer vision code? I always wonder if it means there are new more efficient models still to be found to copy from nature, or if the model ends up not being the most efficient and just the result of evolution.
- visarga 9y ago
  It's not more efficient than CNNs - actually, CNNs are one of the most successful ways to do neural nets. Yann LeCun should be proud.
- ragebol 9y ago
  Sort-of: https://cmusatyalab.github.io/openface/#overview https://cmusatyalab.github.io/openface/#overview
ragebol 9y ago
If I understand this correctly, this works similar to an embedding in e.g. deep learning: faces are represented by high-dimensional vectors. Reading the Cell article on this [0] I couldn't help to see the similarities with OpenFace [1]. [0] http://www.cell.com/cell/fulltext/S0092-8674(17)30538-X http://www.cell.com/cell/fulltext/S0092-8674(17)30538-X [1] https://cmusatyalab.github.io/openface/#overview https://cmusatyalab.github.io/openface/#overview
- supernumerary 9y ago
  Interestingly in the article, they try to differentiate the system from machine learning: Advances in machine learning have been made by training a computerized mimic of a neural network on a given task. Though the networks are successful, they are also a black box because it is hard to reconstruct how they achieve their result. “This has given neuroscience a sense of pessimism that the brain is similarly a black box,” she said. “Our paper provides a counterexample. We’re recording from neurons at the highest stage of the visual system and can see that there’s no black box. My bet is that that will be true throughout the brain.”
  - taneq 9y ago
    At this stage I don't think "black box" is a very fair description. We now understand a fair bit about how artificial neural nets encode information and calculate things. And what we don't understand, we can still see and study the processes.
    - bonzini 9y ago
      We cannot be entirely sure of the behavior in any situation that is sufficiently different from anything that was presented during training. You can't necessarily expect a "common sense" response in that case, and in this sense the neural networks are black boxes.
      - taneq 9y ago
        The same applies to any complex system. 'Black box' means 'not inspectable', not 'anything which doesn't have predictable behaviour over all conceivable inputs'.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
iandanforth 9y ago
Key sentence - "the correct choice of face space axes is critical for achieving a simple explanation of face cells’ responses." They did PCA over two sets of metrics, taking the top 25 components from each set and then combined that into a 50d space. Using these dimensions and measured responses to fit a model resulted in explaining 57% of variance in real cell firing rates. (Much better than other models including a 5 layer CNN). This is pretty cool. I'd like to see a follow-up where the chosen dimensions were further refined using something a bit more iterative that an arbitrary PCA cutoff. Also I really want to know what eye motion was present during each trial. This paper presents a very "instantaneous" recognition perspective and doesn't talk about integration over time or the impact of sequential perception of face components on recognition. (Eg an upside-down face is hard to recognize because your gaze has to move up from the eyes to see the mouth which is a sequence rarely encountered in the real world)
- gumby 9y ago
  > Also I really want to know what eye motion was present during each trial. Incredibly important point. Looking at the physiology of the eye and the early visual system all together it's not clear that we can see anything without motion: object motion or saccade (saccade appears to be fundamental). Start with Lettvin's famous frog's eye paper. More interestingly, it does appear that recognition of a 2D image may be a complex, abstract learned behavor, while 3D recognition is at its core innate. And that writing (well, drawing) preceded reading.
  - stinos 9y ago
    See the methods: as usual for visual fixattion experiments with monkeys, the subjects were trained to maintain fixation, fixation window being only a part of the image size: stimulus size spanned 5.7 degrees. The fixation spot size was 0.2 degrees in diameter and the fixation window was a square with the diameter of 2.5 degrees. 2.5 is relatively large in my experience, might be due to lots of noise on the signal, but anyway there won't be true saccades (well, there shouldn't be, parts of recorded data wehere with saccades outside of that fixation window should get rejected). Probably microsaccades were present, but those aren't mentioned.
mzitelli 9y ago
Amazing results, it is incredible to see that our brains do the same process as CNNs, encoding information using multiple layers of neurons to extract features. This makes me think that consciousness could be just an extreme high-level temporal representation of our own senses.
- fatjokes 9y ago
  I'm not certain, but I don't think that's a coincidence. I.e. convnets were inspired by the neural visual system.
curun1r 9y ago
I remember reading about a study [1] that showed that humans recognize faces based on how similar they are to the faces of their parents. It's well known that humans are more easily able to differentiate faces within our own races. But what the study did was look at people who were adopted by parents of a different race. Those people were more easily able to differentiate faces of people of the same race as their adopted parents and had difficulty differentiating faces of people of their own race. The inference is that we actually store/recognize facial deltas, not full facial images. I'm curious how this study would explain or contradict the results of that study. Also, were the monkeys raised by monkey parents or human scientists? Monkeys that were allowed to imprint on humans might be more similar to humans and, yet, unrepresentative of monkeys. [1] I think it was https://www.ncbi.nlm.nih.gov/pubmed/15943669 https://www.ncbi.nlm.nih.gov/pubmed/15943669
- anothercomment 9y ago
  Hm, I always guessed that the brain simply goes by a kind of "maximum information gain" approach, picking out the features that tend stick out most to distinguish things. If unused to, skin color sticks out extremely, so the signal it generates would drown out the other signals. But that is just a guess and might also go against the findings of the article, which I haven't fully understood yet. In any case it seems to me that the brain is optimized for processing certain features, as apparently there are people who are unable to recognize faces (face blindness).
DanielleMolloy 9y ago
This paper is highly exciting for anybody working on the neural code and so-called encoding models. Only few days ago there was a similar study reading faces from human brains by trying to construct a latent space: https://arxiv.org/abs/1705.07109 https://arxiv.org/abs/1705.07109 https://twitter.com/ccnlab/status/866548346751725568 https://twitter.com/ccnlab/status/866548346751725568 (animation, over time more dimensions from this latent space (afaik PCA components) are added) Given the limitations of fMRI (we can not do single cell recordings in human brains) the results are not as accurate, but to my knowledge this is the best we can do in humans so far.
lexicality 9y ago
"It is a remarkable advance to have identified the dimensions used by the primate brain to decode faces, he added — and impressive that the researchers were able to reconstruct from neural signals the face a monkey is looking at." "These dimensions create a mental “face space” in which an infinite number of faces can be recognized. There is probably an average face, or something like it, at the origin, and the brain measures the deviation from this base." "Dr. Tsao said she was particularly impressed to find she could design a whole series of faces that a given face cell would not respond to, because they lacked its preferred combination of dimensions. This ruled out a possible alternative method of face identification: that the face cells were comparing incoming images with a set of standard reference faces and looking for differences." I'm surprised that they didn't attempt to generate a face with exactly 0 on all dimensions. It would be fascinating to know what the most memorable face looks like - and if it's different per-brain. (Presumably it is monkey shaped!)
linux2647 9y ago
I wonder if this is related to why we recognize faces in nonliving objects such as cars.
daxfohl 9y ago
I'm skeptical. Like "faked results" skeptical. Crime witness studies show that most humans can't reproduce another human face that accurately. So-so when the face is at least of the same race, but when of a different race it's a coin flip as to whether they can even recognize it. (That said, I've only heard this on various TV shows, never seen actual research, so the presumption could be wrong). How can primates do so much better with an entirely different species? Or, not even primates, but some AI going through primate neural signals?
- tmalsburg2 9y ago
  Very good point, and yes there's a lot of research showing that humans are terrible at recognizing faces from other races. No idea why you're being downvoted.
folli 9y ago
How are these macaques able to so finely differentiate faces of a different species? I'm pretty sure I wouldn't be able to differentiate many macaque faces from each other.

197 points by pk2200 9y ago

Here's the paper: http://www.cell.com/cell/pdf/S0092-8674(17)30538-X.pdf http://www.cell.com/cell/pdf/S0092-8674(17)30538-X.pdf

paulfrancisco 9y ago
All the faces seem computer generated. It would have been nice if they had used celebrity faces we can all recognize and see what their system comes up with.
- iandanforth 9y ago
  They are real faces that are morphed along one of the 50 axes they came up with. They look less real because it's standard practice to exclude hair and shoulders and backgrounds from face databases (even if this is totally unrealistic it helps isolated the system under study)
- rangibaby 9y ago
  You can see some of the faces your brain comes up with here: https://youtu.be/VT9i99D_9gI https://youtu.be/VT9i99D_9gI

gech 9y ago

Can this model be translated into computer vision code? I always wonder if it means there are new more efficient models still to be found to copy from nature, or if the model ends up not being the most efficient and just the result of evolution.

visarga 9y ago
It's not more efficient than CNNs - actually, CNNs are one of the most successful ways to do neural nets. Yann LeCun should be proud.
ragebol 9y ago
Sort-of: https://cmusatyalab.github.io/openface/#overview https://cmusatyalab.github.io/openface/#overview

ragebol 9y ago

If I understand this correctly, this works similar to an embedding in e.g. deep learning: faces are represented by high-dimensional vectors. Reading the Cell article on this [0] I couldn't help to see the similarities with OpenFace [1]. [0] http://www.cell.com/cell/fulltext/S0092-8674(17)30538-X http://www.cell.com/cell/fulltext/S0092-8674(17)30538-X [1] https://cmusatyalab.github.io/openface/#overview https://cmusatyalab.github.io/openface/#overview

supernumerary 9y ago
Interestingly in the article, they try to differentiate the system from machine learning: Advances in machine learning have been made by training a computerized mimic of a neural network on a given task. Though the networks are successful, they are also a black box because it is hard to reconstruct how they achieve their result. “This has given neuroscience a sense of pessimism that the brain is similarly a black box,” she said. “Our paper provides a counterexample. We’re recording from neurons at the highest stage of the visual system and can see that there’s no black box. My bet is that that will be true throughout the brain.”
- taneq 9y ago
  At this stage I don't think "black box" is a very fair description. We now understand a fair bit about how artificial neural nets encode information and calculate things. And what we don't understand, we can still see and study the processes.
  - bonzini 9y ago
    We cannot be entirely sure of the behavior in any situation that is sufficiently different from anything that was presented during training. You can't necessarily expect a "common sense" response in that case, and in this sense the neural networks are black boxes.
    - taneq 9y ago
      The same applies to any complex system. 'Black box' means 'not inspectable', not 'anything which doesn't have predictable behaviour over all conceivable inputs'.

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

iandanforth 9y ago

Key sentence - "the correct choice of face space axes is critical for achieving a simple explanation of face cells’ responses." They did PCA over two sets of metrics, taking the top 25 components from each set and then combined that into a 50d space. Using these dimensions and measured responses to fit a model resulted in explaining 57% of variance in real cell firing rates. (Much better than other models including a 5 layer CNN). This is pretty cool. I'd like to see a follow-up where the chosen dimensions were further refined using something a bit more iterative that an arbitrary PCA cutoff. Also I really want to know what eye motion was present during each trial. This paper presents a very "instantaneous" recognition perspective and doesn't talk about integration over time or the impact of sequential perception of face components on recognition. (Eg an upside-down face is hard to recognize because your gaze has to move up from the eyes to see the mouth which is a sequence rarely encountered in the real world)

gumby 9y ago
> Also I really want to know what eye motion was present during each trial. Incredibly important point. Looking at the physiology of the eye and the early visual system all together it's not clear that we can see anything without motion: object motion or saccade (saccade appears to be fundamental). Start with Lettvin's famous frog's eye paper. More interestingly, it does appear that recognition of a 2D image may be a complex, abstract learned behavor, while 3D recognition is at its core innate. And that writing (well, drawing) preceded reading.
- stinos 9y ago
  See the methods: as usual for visual fixattion experiments with monkeys, the subjects were trained to maintain fixation, fixation window being only a part of the image size: stimulus size spanned 5.7 degrees. The fixation spot size was 0.2 degrees in diameter and the fixation window was a square with the diameter of 2.5 degrees. 2.5 is relatively large in my experience, might be due to lots of noise on the signal, but anyway there won't be true saccades (well, there shouldn't be, parts of recorded data wehere with saccades outside of that fixation window should get rejected). Probably microsaccades were present, but those aren't mentioned.

mzitelli 9y ago

Amazing results, it is incredible to see that our brains do the same process as CNNs, encoding information using multiple layers of neurons to extract features. This makes me think that consciousness could be just an extreme high-level temporal representation of our own senses.

fatjokes 9y ago
I'm not certain, but I don't think that's a coincidence. I.e. convnets were inspired by the neural visual system.

curun1r 9y ago

I remember reading about a study [1] that showed that humans recognize faces based on how similar they are to the faces of their parents. It's well known that humans are more easily able to differentiate faces within our own races. But what the study did was look at people who were adopted by parents of a different race. Those people were more easily able to differentiate faces of people of the same race as their adopted parents and had difficulty differentiating faces of people of their own race. The inference is that we actually store/recognize facial deltas, not full facial images. I'm curious how this study would explain or contradict the results of that study. Also, were the monkeys raised by monkey parents or human scientists? Monkeys that were allowed to imprint on humans might be more similar to humans and, yet, unrepresentative of monkeys. [1] I think it was https://www.ncbi.nlm.nih.gov/pubmed/15943669 https://www.ncbi.nlm.nih.gov/pubmed/15943669

anothercomment 9y ago
Hm, I always guessed that the brain simply goes by a kind of "maximum information gain" approach, picking out the features that tend stick out most to distinguish things. If unused to, skin color sticks out extremely, so the signal it generates would drown out the other signals. But that is just a guess and might also go against the findings of the article, which I haven't fully understood yet. In any case it seems to me that the brain is optimized for processing certain features, as apparently there are people who are unable to recognize faces (face blindness).

DanielleMolloy 9y ago

This paper is highly exciting for anybody working on the neural code and so-called encoding models. Only few days ago there was a similar study reading faces from human brains by trying to construct a latent space: https://arxiv.org/abs/1705.07109 https://arxiv.org/abs/1705.07109 https://twitter.com/ccnlab/status/866548346751725568 https://twitter.com/ccnlab/status/866548346751725568 (animation, over time more dimensions from this latent space (afaik PCA components) are added) Given the limitations of fMRI (we can not do single cell recordings in human brains) the results are not as accurate, but to my knowledge this is the best we can do in humans so far.

lexicality 9y ago

"It is a remarkable advance to have identified the dimensions used by the primate brain to decode faces, he added — and impressive that the researchers were able to reconstruct from neural signals the face a monkey is looking at." "These dimensions create a mental “face space” in which an infinite number of faces can be recognized. There is probably an average face, or something like it, at the origin, and the brain measures the deviation from this base." "Dr. Tsao said she was particularly impressed to find she could design a whole series of faces that a given face cell would not respond to, because they lacked its preferred combination of dimensions. This ruled out a possible alternative method of face identification: that the face cells were comparing incoming images with a set of standard reference faces and looking for differences." I'm surprised that they didn't attempt to generate a face with exactly 0 on all dimensions. It would be fascinating to know what the most memorable face looks like - and if it's different per-brain. (Presumably it is monkey shaped!)

linux2647 9y ago

I wonder if this is related to why we recognize faces in nonliving objects such as cars.

daxfohl 9y ago

I'm skeptical. Like "faked results" skeptical. Crime witness studies show that most humans can't reproduce another human face that accurately. So-so when the face is at least of the same race, but when of a different race it's a coin flip as to whether they can even recognize it. (That said, I've only heard this on various TV shows, never seen actual research, so the presumption could be wrong). How can primates do so much better with an entirely different species? Or, not even primates, but some AI going through primate neural signals?

tmalsburg2 9y ago
Very good point, and yes there's a lot of research showing that humans are terrible at recognizing faces from other races. No idea why you're being downvoted.

folli 9y ago

How are these macaques able to so finely differentiate faces of a different species? I'm pretty sure I wouldn't be able to differentiate many macaque faces from each other.