4 ms·

Reconstruction Of Visual Images Using Brain Activity Patterns

78 points by jcr 14y ago

jcr 14y ago
The submitted link is some of the background work for the recent news about decoding fMRI signals during dreaming into actual images. A paper on this work was released today on the journal Science (sciencemag.org) but I've been unable to find a free copy. Some of the videos in the last link below are really interesting. http://www.wired.com/wiredscience/2013/04/dream-decoder http://www.wired.com/wiredscience/2013/04/dream-decoder http://medicalxpress.com/news/2013-04-scientists-peek.html http://medicalxpress.com/news/2013-04-scientists-peek.html http://www.sciencemag.org/content/340/6128/21.summary http://www.sciencemag.org/content/340/6128/21.summary http://www.sciencemag.org/content/early/2013/04/03/science.1234330/suppl/DC1 http://www.sciencemag.org/content/early/2013/04/03/science.1...
- yk 14y ago
  Can you summarize how the videos in the last link were created?
  - jcr 14y ago
    Unfortunately no. The details are most likely in the journal article (sciencemag.org), but I've been unable to find a free copy of it. (I'm really trying to resist ranting about the "Evil Overlords" of academic publishing ;)
  - creamyhorror 14y ago
    According to the abstract: Here, we present a neural decoding approach in which machine learning models predict the contents of visual imagery during the sleep onset period given measured brain activity, by discovering links between human fMRI patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. So it sounds like the fMRI patterns are associated with words via verbal reports from humans, and the video is simply generated from image databases by using those words. In other words, they classify the fMRI patterns based on how well they correspond to various words (measured while the subjects are awake and giving verbal descriptions of images they're looking at), and then the program measures the subjects' fMRI patterns while they're dreaming, determines the strongest 'word signals' in the patterns, and selects images from a photo database corresponding to those words and assembles them into a video based on how strong the similarity score for each word is at each moment in time. It's interesting, but the video seems like more of a "reenactment" of a dream than a prediction. It doesn't look like you can infer actions and occurrences within a dream from this. But this is all just my interpretation. edit: This is a fundamentally different approach from the one used in other research (e.g. http://www.youtube.com/watch?v=nsjDnYxJ0bo http://www.youtube.com/watch?v=nsjDnYxJ0bo mentioned elsewhere) in which image data is statistically regressed on fMRI patterns generated while the subjects watch those images. That approach allows for a much more precise reenactment/prediction of the actual images being dreamed.
runn1ng 14y ago
This is one year and something old and returns better results, in my opinion. http://www.youtube.com/watch?v=nsjDnYxJ0bo http://www.youtube.com/watch?v=nsjDnYxJ0bo On the other hand, they presented only the best results in that video.
- jcr 14y ago
  Thanks. Looks interesting (if you pardon the pun). I think the big difference in the approaches revolve around this claim, "Using this method, we can reconstruct arbitrary visual images, including geometric figures and alphabet letters, which are not used to train the decoders." versus > [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction. It seems gallantlab.org work is using known images/video along with the subject in the fMRI (magnet) being awake. Decoding arbitrary images seems to be the main advancement being leveraged in the dream decoding research.
  - runn1ng 14y ago
    That makes sense, because in the gallantlab video, you can clearly see the residual images from the previous videos (letters, different shapes).
- miga 14y ago
  Best results maybe, but both MRI resolution and frame acquisition times seem to improve fast: http://www.youtube.com/watch?v=uTOhDqhCKQs http://www.youtube.com/watch?v=uTOhDqhCKQs http://www.youtube.com/watch?v=rtBd1y0_Pvo http://www.youtube.com/watch?v=rtBd1y0_Pvo
Gravityloss 14y ago
Cool! You can also see the after images (black square after white square).
its_so_on 14y ago
It clearly says, "the prediction is just categorical and is not an ‘image’ itself." From the article: >Neural decoding is performed using pattern recognition programs, also known as “decoders,” that learn statistical relationships between several pre-specified visual images and the corresponding brain activity patterns. Although this approach can predict which stimulus is presented among a set of pre-specified categories (e.g., horizontal or vertical grating), the prediction is just categorical and is not an ‘image’ itself. In other words, this similar to saying, "Reconstruction of password using CPU activity" and then being able to tell which of a pre-specified list of passwords was used, based on clocking the CPU. This is not exactly reconstruction.
Aardwolf 14y ago
Reminds me of the Dreamcatcher 3000: http://pbfcomics.com/160/ http://pbfcomics.com/160/

78 points by jcr 14y ago

jcr 14y ago

The submitted link is some of the background work for the recent news about decoding fMRI signals during dreaming into actual images. A paper on this work was released today on the journal Science (sciencemag.org) but I've been unable to find a free copy. Some of the videos in the last link below are really interesting. http://www.wired.com/wiredscience/2013/04/dream-decoder http://www.wired.com/wiredscience/2013/04/dream-decoder http://medicalxpress.com/news/2013-04-scientists-peek.html http://medicalxpress.com/news/2013-04-scientists-peek.html http://www.sciencemag.org/content/340/6128/21.summary http://www.sciencemag.org/content/340/6128/21.summary http://www.sciencemag.org/content/early/2013/04/03/science.1234330/suppl/DC1 http://www.sciencemag.org/content/early/2013/04/03/science.1...

yk 14y ago
Can you summarize how the videos in the last link were created?
- jcr 14y ago
  Unfortunately no. The details are most likely in the journal article (sciencemag.org), but I've been unable to find a free copy of it. (I'm really trying to resist ranting about the "Evil Overlords" of academic publishing ;)
- creamyhorror 14y ago
  According to the abstract: Here, we present a neural decoding approach in which machine learning models predict the contents of visual imagery during the sleep onset period given measured brain activity, by discovering links between human fMRI patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. So it sounds like the fMRI patterns are associated with words via verbal reports from humans, and the video is simply generated from image databases by using those words. In other words, they classify the fMRI patterns based on how well they correspond to various words (measured while the subjects are awake and giving verbal descriptions of images they're looking at), and then the program measures the subjects' fMRI patterns while they're dreaming, determines the strongest 'word signals' in the patterns, and selects images from a photo database corresponding to those words and assembles them into a video based on how strong the similarity score for each word is at each moment in time. It's interesting, but the video seems like more of a "reenactment" of a dream than a prediction. It doesn't look like you can infer actions and occurrences within a dream from this. But this is all just my interpretation. edit: This is a fundamentally different approach from the one used in other research (e.g. http://www.youtube.com/watch?v=nsjDnYxJ0bo http://www.youtube.com/watch?v=nsjDnYxJ0bo mentioned elsewhere) in which image data is statistically regressed on fMRI patterns generated while the subjects watch those images. That approach allows for a much more precise reenactment/prediction of the actual images being dreamed.

runn1ng 14y ago

This is one year and something old and returns better results, in my opinion. http://www.youtube.com/watch?v=nsjDnYxJ0bo http://www.youtube.com/watch?v=nsjDnYxJ0bo On the other hand, they presented only the best results in that video.

jcr 14y ago
Thanks. Looks interesting (if you pardon the pun). I think the big difference in the approaches revolve around this claim, "Using this method, we can reconstruct arbitrary visual images, including geometric figures and alphabet letters, which are not used to train the decoders." versus > [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction. It seems gallantlab.org work is using known images/video along with the subject in the fMRI (magnet) being awake. Decoding arbitrary images seems to be the main advancement being leveraged in the dream decoding research.
- runn1ng 14y ago
  That makes sense, because in the gallantlab video, you can clearly see the residual images from the previous videos (letters, different shapes).
miga 14y ago
Best results maybe, but both MRI resolution and frame acquisition times seem to improve fast: http://www.youtube.com/watch?v=uTOhDqhCKQs http://www.youtube.com/watch?v=uTOhDqhCKQs http://www.youtube.com/watch?v=rtBd1y0_Pvo http://www.youtube.com/watch?v=rtBd1y0_Pvo

Gravityloss 14y ago

Cool! You can also see the after images (black square after white square).

its_so_on 14y ago

It clearly says, "the prediction is just categorical and is not an ‘image’ itself." From the article: >Neural decoding is performed using pattern recognition programs, also known as “decoders,” that learn statistical relationships between several pre-specified visual images and the corresponding brain activity patterns. Although this approach can predict which stimulus is presented among a set of pre-specified categories (e.g., horizontal or vertical grating), the prediction is just categorical and is not an ‘image’ itself. In other words, this similar to saying, "Reconstruction of password using CPU activity" and then being able to tell which of a pre-specified list of passwords was used, based on clocking the CPU. This is not exactly reconstruction.

Aardwolf 14y ago

Reminds me of the Dreamcatcher 3000: http://pbfcomics.com/160/ http://pbfcomics.com/160/