6 ms·

Yale researchers reconstruct facial images locked in a viewer’s mind

159 points by turing 13y ago

freehunter 13y ago
My first thought of this would be its use in constructing a image of a wanted criminal, as a way to replace police sketch artists. When I viewed their image, they're incredibly close, but I don't think they're quite there. I'm really looking forward to seeing this improve as they've stated it will. I thought the woman looked close enough to be able to identify, but the man was not. Still, very impressive work.
- scoot 13y ago
  My first thought of this would be its use in constructing a image of a wanted criminal, as a way to replace police sketch artists. They're reconstructing faces the subject is viewing, not remembering. To replace police sketch artists, the criminal would have to be present...
  - freehunter 13y ago
    Ah, I guess I misread. Thanks for the correction.
- ballard 13y ago
  The interesting bit would be reconstructing the right face memory rather than one of the interviewers. Actually, there's an obvious scifi plot: investigator is a serial killer but excluded from memory extraction because the witness just saw them (again).
WildUtah 13y ago
Soon we will be able to finally do away with that hoary old libertarian canard that the state cannot judge you and punish you for what you think inside your own head. Just imagine how harmony and true equality will blossom then!
bttf 13y ago
We now have read access to a person's mind and its visual information when dealing with faces. Naturally, write access cannot be far away ...
- rquantz 13y ago
  The faces got into their brains in the first place because we already have write access. Writing to the brain is easy -- it's reading what we've put there that is difficult.
  - a-priori 13y ago
    It is possible to reconstruct an image by measuring the activation of V1 (primary visual cortex). The same technique works whether the image is currently on the visual field, the person simply recalls an image from memory.
- gregcrv 13y ago
  We already have write access: Subliminal stimuli and it has been used for a while now ! http://en.wikipedia.org/wiki/Subliminal_stimuli http://en.wikipedia.org/wiki/Subliminal_stimuli
- a-priori 13y ago
  Actually, we're already there. It's called a visual prosthesis. http://en.wikipedia.org/wiki/Visual_prosthesis http://en.wikipedia.org/wiki/Visual_prosthesis
- cookingrobot 13y ago
  Writing to a person's visual system is easy - just show them a picture.
- deleted 13y ago
  [deleted]
chch 13y ago
Although it's hard to tell from the images presented with the article, the face generation looks like it could be similar to the techniques used in Nishimoto et al., 2011, which used a similar library of learned brain responses, though for movie trailers: http://www.youtube.com/watch?v=nsjDnYxJ0bo http://www.youtube.com/watch?v=nsjDnYxJ0bo Their particular process is described in the YouTube caption: The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows: [1] Record brain activity while the subject watches several hours of movie trailers. [2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured. (For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.) [3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions. [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction. With the actual paper here: http://www.cell.com/current-biology/retrieve/pii/S0960982211009377 http://www.cell.com/current-biology/retrieve/pii/S0960982211...
- anigbrowl 13y ago
  Forgot about this, thanks for re-posting.
  - JackFr 13y ago
    Thanks for posting that. I count myself as enormously skeptical of TFA research, but that paper appears to be quite good. I may need to re-evaluate my biases. On the other hand, this is presented as in the press release as mind reading, but the reality is more like trying design something similar to a cochlear implant.
TrainedMonkey 13y ago
You know maybe people wearing foil hats were on to something after all. In all seriousness this is a great advance in neuroscience that would help understand many things about brain. On the other hand, potential for misuse is enormous. Can you even prove you had been interrogated if such a device is used on you?
- bashinator 13y ago
  I think it's going to be quite a long time before fMRI machines get to the point that you don't know one is being used on you. Room-temperature superconductors are probably a requirement.
spikels 13y ago
Too bad the reconstructed faces don't look anything like the presented faces and I'm sure these two example are some of the best results. I suspect the algorithm always outputs some face generated by a paramaterized face model (neutral net based?). Therefore even random output would generate a face. Then with some "tuning" and a little wishful thinking you might convince yourself this works. Am I being too skeptical?
- friendcomputer 13y ago
  From TFA: "Working with funding from the Yale Provost’s office, Cowen and post doctoral researcher Brice Kuhl, now an assistant professor at New York University, showed six subjects 300 different “training” faces while undergoing fMRI scans. They used the data to create a sort of statistical library of how those brains responded to individual faces. They then showed the six subjects new sets of faces while they were undergoing scans. Taking that fMRI data alone, researchers used their statistical library to reconstruct the faces their subjects were viewing." So yes, it will always output something like a face. It's more like they are using the FMRI to select among preset options. It's still potentially a great result, but we need more detail than this article provides.
- JackFr 13y ago
  I would say not skeptical enough. Generating a result which looks like some combination of the inputs, has a little bit of 'Wow' factor if you just look at the pictures and don't think too hard about it. But ultimately its not an exercise in which they can be wrong, and since the image returned is always going to be a face, they'll always kind of be right. An impressive result would be if they trained their system on the the 300 pictures, and then we're reliably able identify which picture the subject was looking at. That would be quantifiable and testable, and I presume the result would expose that this is all nonsense.
  - 001sky 13y ago
    "Since the image returned is always going to be a face, they'll always kind of be right" So, non-falsifiable...?
- 13y ago
tokenadult 13y ago
Cool result, but where is a publication showing the faces that were actually tested and "reconstructed"? Many, many submissions to HN (like this one) are press releases, and press releases are well known for spinning preliminary research findings beyond all recognition. This has been commented on in the PhD comic "The Science News Cycle,"[1] which only exaggerates the process a very little. More serious commentary in the edited group blog post "Related by coincidence only? University and medical journal press releases versus journal articles"[2] points to the same danger of taking press releases (and news aggregator website articles based solely on press releases) too seriously. I look forward to seeing how this finding develops as it is commented on and reviewed by other researchers in peer-reviewed publications and attempts to replicate the finding. The most sure and certain finding of any preliminary study will be that more research is needed. Disappointingly often, preliminary findings don't lead to further useful discoveries in science, because the preliminary findings are flawed. If the technique reported here can generalize at sufficiently low expense, it could lead to a lot of insight into the workings of the known-to-be complicated neural networks of the human brain used for recognizing faces. A useful follow-up link for any discussion of a report on a research result like the one kindly submitted here is the article "Warning Signs in Experimental Design and Interpretation"[3] by Peter Norvig, director of research at Google, on how to interpret scientific research. Check each news story you read for how many of the important issues in interpreting research are NOT discussed in the story. [1] http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174 [2] http://www.sciencebasedmedicine.org/index.php/related-by-coincidence-only-journal-press-releases-versus-journal-articles/ http://www.sciencebasedmedicine.org/index.php/related-by-coi... [3] http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html
- anigbrowl 13y ago
  Paper accessible via http://www.sciencedirect.com/science/article/pii/S1053811914001633?_rdoc=1&_fmt=high&_origin=ihub&_docanchor=&md5=9ffa87934275edd7180b52f5e973f002 http://www.sciencedirect.com/science/article/pii/S1053811914... if you have an academic login. Your points on experiemtnal design are good but I feel you're grandstanding a bit here; the reason the press release is posted is because it's so hard to get access to the text of scientific papers thanks to the publishing industry, so citing essays about experimental design ends up casting implicit aspersions on the quality of the authors' work, as if the paper would be more publicly available but for some reticence on their part. I would like to see drastic changes to the journal publishing model, but I don't consider article inaccessibility to be correlated with flaws in the findings. To bring up the latter out of frustration over the former is poisoning the well of debate. Incidentally, I found the paper in under 30 seconds by searching for the journal name, the senior author, and a few technical terms, all of which were in the press release and which gave me a correct first search result. It would be nice if the press release also contained a DOI link and other identifying information, but I can't really blame press departments for supplying journalists with the information they want and omitting that which most of them don't.
aasarava 13y ago
Let's stop with the "mind reading" warnings before they get too far out of hand and consider what's really happening: Six subjects were shown a "training" corpus of images first. Then shown new images. By comparing the subjects' responses to the new images, the software in the study presumably did its best to create composite images by pulling from the corpus. So this raises many questions: How diverse were the faces in the training corpus? How close were the new images to those in the corpus? When you're looking at hundreds of images to train the machine, are you also unknowingly being trained to think about images in a certain way? What happens when you try to recreate faces based on the fMRI responses of subjects who didn't contribute to the initial training set? The implications of the last question are pretty interesting . If different people have different brain responses to looking at the same image, does that help us begin to understand why you and I can be attracted to different types of people? Does it help begin to explain why two people can experience the same event but walk away with two completely different interpretations?
- JackFr 13y ago
  "If use terms like 'mind reading' in our press release we can get picked up by a major news outlet, despite having no real notable scientific result." Hypothesis confirmed!
systematical 13y ago
Great, now I can figure out who I slept with after blacking out.
electrichead 13y ago
I found it interesting that the researcher thought that there was no possibility of receiving external funding for something like this. I would have thought the opposite. In fact I can think of a bunch of companies who would be only too happy to throw money at so ething like this.
notastartup 13y ago
so...we can eventually create games and movies by imagining it without the need to do it by hand and input it into a computer
- Houshalter 13y ago
  Possibly. One thing I'm worried about is that our brains might not be good generative models. We remember the high level details, not a pixel by pixel map of what we imagine. That's probably why humans aren't naturally good at drawing. We can imagine a picture but trying to create an image on paper that matches our imagination is quite difficult.
  - auxon0 13y ago
    True. However, if we are able to elevate this type of technology to that level, people will train and be trained, to focus on such detail and learn to control the output via the feedback loop. At first you'd probably "draw" like a baby, but with enough practice, some would become true artists, and the average somewhere in the middle, but things would be recognizable and tada, telepathy.
spektom 13y ago
I wonder whether this will work with different person, not the one who participated in the machine learning process. Someone, who has different brain activities when the same faces from the training set are shown. Is that possible that our brains analyze the seen differently?
cma 13y ago
This one is crazy: https://www.youtube.com/watch?v=SjbSEjOJL3U https://www.youtube.com/watch?v=SjbSEjOJL3U (Mary Lou Jepsen on basically recreating video of what a person is watching through fMRI)

159 points by turing 13y ago

My first thought of this would be its use in constructing a image of a wanted criminal, as a way to replace police sketch artists. When I viewed their image, they're incredibly close, but I don't think they're quite there. I'm really looking forward to seeing this improve as they've stated it will. I thought the woman looked close enough to be able to identify, but the man was not. Still, very impressive work.

scoot 13y ago
My first thought of this would be its use in constructing a image of a wanted criminal, as a way to replace police sketch artists. They're reconstructing faces the subject is viewing, not remembering. To replace police sketch artists, the criminal would have to be present...
- freehunter 13y ago
  Ah, I guess I misread. Thanks for the correction.
ballard 13y ago
The interesting bit would be reconstructing the right face memory rather than one of the interviewers. Actually, there's an obvious scifi plot: investigator is a serial killer but excluded from memory extraction because the witness just saw them (again).

WildUtah 13y ago

Soon we will be able to finally do away with that hoary old libertarian canard that the state cannot judge you and punish you for what you think inside your own head. Just imagine how harmony and true equality will blossom then!

bttf 13y ago

We now have read access to a person's mind and its visual information when dealing with faces. Naturally, write access cannot be far away ...

rquantz 13y ago
The faces got into their brains in the first place because we already have write access. Writing to the brain is easy -- it's reading what we've put there that is difficult.
- a-priori 13y ago
  It is possible to reconstruct an image by measuring the activation of V1 (primary visual cortex). The same technique works whether the image is currently on the visual field, the person simply recalls an image from memory.
gregcrv 13y ago
We already have write access: Subliminal stimuli and it has been used for a while now ! http://en.wikipedia.org/wiki/Subliminal_stimuli http://en.wikipedia.org/wiki/Subliminal_stimuli
a-priori 13y ago
Actually, we're already there. It's called a visual prosthesis. http://en.wikipedia.org/wiki/Visual_prosthesis http://en.wikipedia.org/wiki/Visual_prosthesis
cookingrobot 13y ago
Writing to a person's visual system is easy - just show them a picture.
deleted 13y ago
[deleted]

chch 13y ago

Although it's hard to tell from the images presented with the article, the face generation looks like it could be similar to the techniques used in Nishimoto et al., 2011, which used a similar library of learned brain responses, though for movie trailers: http://www.youtube.com/watch?v=nsjDnYxJ0bo http://www.youtube.com/watch?v=nsjDnYxJ0bo Their particular process is described in the YouTube caption: The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows: [1] Record brain activity while the subject watches several hours of movie trailers. [2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured. (For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.) [3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions. [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction. With the actual paper here: http://www.cell.com/current-biology/retrieve/pii/S0960982211009377 http://www.cell.com/current-biology/retrieve/pii/S0960982211...

anigbrowl 13y ago
Forgot about this, thanks for re-posting.
- JackFr 13y ago
  Thanks for posting that. I count myself as enormously skeptical of TFA research, but that paper appears to be quite good. I may need to re-evaluate my biases. On the other hand, this is presented as in the press release as mind reading, but the reality is more like trying design something similar to a cochlear implant.

TrainedMonkey 13y ago

You know maybe people wearing foil hats were on to something after all. In all seriousness this is a great advance in neuroscience that would help understand many things about brain. On the other hand, potential for misuse is enormous. Can you even prove you had been interrogated if such a device is used on you?

bashinator 13y ago
I think it's going to be quite a long time before fMRI machines get to the point that you don't know one is being used on you. Room-temperature superconductors are probably a requirement.

spikels 13y ago

Too bad the reconstructed faces don't look anything like the presented faces and I'm sure these two example are some of the best results. I suspect the algorithm always outputs some face generated by a paramaterized face model (neutral net based?). Therefore even random output would generate a face. Then with some "tuning" and a little wishful thinking you might convince yourself this works. Am I being too skeptical?

friendcomputer 13y ago
From TFA: "Working with funding from the Yale Provost’s office, Cowen and post doctoral researcher Brice Kuhl, now an assistant professor at New York University, showed six subjects 300 different “training” faces while undergoing fMRI scans. They used the data to create a sort of statistical library of how those brains responded to individual faces. They then showed the six subjects new sets of faces while they were undergoing scans. Taking that fMRI data alone, researchers used their statistical library to reconstruct the faces their subjects were viewing." So yes, it will always output something like a face. It's more like they are using the FMRI to select among preset options. It's still potentially a great result, but we need more detail than this article provides.
JackFr 13y ago
I would say not skeptical enough. Generating a result which looks like some combination of the inputs, has a little bit of 'Wow' factor if you just look at the pictures and don't think too hard about it. But ultimately its not an exercise in which they can be wrong, and since the image returned is always going to be a face, they'll always kind of be right. An impressive result would be if they trained their system on the the 300 pictures, and then we're reliably able identify which picture the subject was looking at. That would be quantifiable and testable, and I presume the result would expose that this is all nonsense.
- 001sky 13y ago
  "Since the image returned is always going to be a face, they'll always kind of be right" So, non-falsifiable...?
13y ago

tokenadult 13y ago

Cool result, but where is a publication showing the faces that were actually tested and "reconstructed"? Many, many submissions to HN (like this one) are press releases, and press releases are well known for spinning preliminary research findings beyond all recognition. This has been commented on in the PhD comic "The Science News Cycle,"[1] which only exaggerates the process a very little. More serious commentary in the edited group blog post "Related by coincidence only? University and medical journal press releases versus journal articles"[2] points to the same danger of taking press releases (and news aggregator website articles based solely on press releases) too seriously. I look forward to seeing how this finding develops as it is commented on and reviewed by other researchers in peer-reviewed publications and attempts to replicate the finding. The most sure and certain finding of any preliminary study will be that more research is needed. Disappointingly often, preliminary findings don't lead to further useful discoveries in science, because the preliminary findings are flawed. If the technique reported here can generalize at sufficiently low expense, it could lead to a lot of insight into the workings of the known-to-be complicated neural networks of the human brain used for recognizing faces. A useful follow-up link for any discussion of a report on a research result like the one kindly submitted here is the article "Warning Signs in Experimental Design and Interpretation"[3] by Peter Norvig, director of research at Google, on how to interpret scientific research. Check each news story you read for how many of the important issues in interpreting research are NOT discussed in the story. [1] http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174 [2] http://www.sciencebasedmedicine.org/index.php/related-by-coincidence-only-journal-press-releases-versus-journal-articles/ http://www.sciencebasedmedicine.org/index.php/related-by-coi... [3] http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html

anigbrowl 13y ago
Paper accessible via http://www.sciencedirect.com/science/article/pii/S1053811914001633?_rdoc=1&_fmt=high&_origin=ihub&_docanchor=&md5=9ffa87934275edd7180b52f5e973f002 http://www.sciencedirect.com/science/article/pii/S1053811914... if you have an academic login. Your points on experiemtnal design are good but I feel you're grandstanding a bit here; the reason the press release is posted is because it's so hard to get access to the text of scientific papers thanks to the publishing industry, so citing essays about experimental design ends up casting implicit aspersions on the quality of the authors' work, as if the paper would be more publicly available but for some reticence on their part. I would like to see drastic changes to the journal publishing model, but I don't consider article inaccessibility to be correlated with flaws in the findings. To bring up the latter out of frustration over the former is poisoning the well of debate. Incidentally, I found the paper in under 30 seconds by searching for the journal name, the senior author, and a few technical terms, all of which were in the press release and which gave me a correct first search result. It would be nice if the press release also contained a DOI link and other identifying information, but I can't really blame press departments for supplying journalists with the information they want and omitting that which most of them don't.

aasarava 13y ago

Let's stop with the "mind reading" warnings before they get too far out of hand and consider what's really happening: Six subjects were shown a "training" corpus of images first. Then shown new images. By comparing the subjects' responses to the new images, the software in the study presumably did its best to create composite images by pulling from the corpus. So this raises many questions: How diverse were the faces in the training corpus? How close were the new images to those in the corpus? When you're looking at hundreds of images to train the machine, are you also unknowingly being trained to think about images in a certain way? What happens when you try to recreate faces based on the fMRI responses of subjects who didn't contribute to the initial training set? The implications of the last question are pretty interesting . If different people have different brain responses to looking at the same image, does that help us begin to understand why you and I can be attracted to different types of people? Does it help begin to explain why two people can experience the same event but walk away with two completely different interpretations?

JackFr 13y ago
"If use terms like 'mind reading' in our press release we can get picked up by a major news outlet, despite having no real notable scientific result." Hypothesis confirmed!

systematical 13y ago

Great, now I can figure out who I slept with after blacking out.

electrichead 13y ago

I found it interesting that the researcher thought that there was no possibility of receiving external funding for something like this. I would have thought the opposite. In fact I can think of a bunch of companies who would be only too happy to throw money at so ething like this.

notastartup 13y ago

so...we can eventually create games and movies by imagining it without the need to do it by hand and input it into a computer

Houshalter 13y ago
Possibly. One thing I'm worried about is that our brains might not be good generative models. We remember the high level details, not a pixel by pixel map of what we imagine. That's probably why humans aren't naturally good at drawing. We can imagine a picture but trying to create an image on paper that matches our imagination is quite difficult.
- auxon0 13y ago
  True. However, if we are able to elevate this type of technology to that level, people will train and be trained, to focus on such detail and learn to control the output via the feedback loop. At first you'd probably "draw" like a baby, but with enough practice, some would become true artists, and the average somewhere in the middle, but things would be recognizable and tada, telepathy.

spektom 13y ago

I wonder whether this will work with different person, not the one who participated in the machine learning process. Someone, who has different brain activities when the same faces from the training set are shown. Is that possible that our brains analyze the seen differently?

cma 13y ago

This one is crazy: https://www.youtube.com/watch?v=SjbSEjOJL3U https://www.youtube.com/watch?v=SjbSEjOJL3U (Mary Lou Jepsen on basically recreating video of what a person is watching through fMRI)