24 ms·

Datasets for Reconstructing Visual Perception from Brain Data

62 points by katsee 7mo ago

mitthrowaway2 7mo ago
Is there a case for non-dystopian applications for such a project, should it succeed? I get that we're all driven by curiosity, and the brain is very mysterious, but at some point I really wonder when scientists will start to taboo projects like this for ethical reasons, just like they currently taboo human cloning.
- td2 7mo ago
  I dont know anything about neurosience But if dreaming is related enouth to visual perception, maybe a dream recorder might be somewhat possible
  - td2 7mo ago
    Edit, reading the github description, which i should have done sooner, hints at dreams.
    - bensyverson 7mo ago
      Every researcher working on this should be required to watch Until the End of the World.
- katsee 7mo ago
  Reconstruction of inner experience - imagery, maybe dreams & hallucinations, perceptual phenomena. Here is a recent study reconstructing the inner perception of optical illusions: https://www.science.org/doi/abs/10.1126/sciadv.adj3906 https://www.science.org/doi/abs/10.1126/sciadv.adj3906 Methods like MVPA (decoding among finite sets of cognitive state classes) are actually widely used for insight in cognitive neuroscience. Ethical concerns are discussed within the field; most papers had explicit ethics sections and discussions long before AI conferences required them for all submissions. In practice these experiments require a participant lying motionless (≈1-2 mm range) in an MRI scanner with controlled gaze and attention for many hours, and even then zero-shot reconstruction is not really possible; the SNR requires many repetitions.
- MarcelOlsz 7mo ago
  Not since Americans imported the Nazis - Von Braun et al.
- aspenmartin 7mo ago
  Dystopian applications are extremely impractical or impossible, this is a tool for neuroscience
  - pixl97 7mo ago
    "This will never be used by bad guys" says the person immediately before their tools are used by bad guys.
    - aspenmartin 7mo ago
      No, it's "this tool cannot be used by bad guys or good guys, but can be used by highly funded labs that do neuroscience". It's something that freaks people out until they gradually learn what is actually involved https://ai.meta.com/blog/brain-ai-image-decoding-meg-magnetoencephalography/ https://ai.meta.com/blog/brain-ai-image-decoding-meg-magneto... [2023] https://ai.meta.com/blog/brain-ai-research-human-communication/ https://ai.meta.com/blog/brain-ai-research-human-communicati... [last year, focuses on decoding text] ^ There's a research team at Meta that studies this. You need an MEG -- thats $2-5M + the shielded room it lives in and the experts that can operate it. EEG doesn't work due to low spatial resolution and how finicky it is to place the electrodes to get a good signal The signals from neurons are just unbelievably tiny and are in an absolute sea of noisy trash. No one is ever going to read your thoughts without your consent (or by wrestling you into a big MEG, in which case you have bigger things to worry about). No one is going to be reading your dreams with any sort of accuracy either.
      - mandolingual 7mo ago
        And computers used to fill a room and require stacks of punch cards to use.
        aspenmartin 7mo ago
        in both cases: physics decides what's possible
smusamashah 7mo ago
There was this research where faces were almost perfectly reconstructed from money's brain signals. How were they able to achieve such perfect recreation from monkey but not even close from human brain. https://www.bbc.co.uk/news/science-environment-40131242 https://www.bbc.co.uk/news/science-environment-40131242
- pixl97 7mo ago
  So, the question I have is, could the reassemble the data from a person who has face blindness? From my rather weak understanding on the subject, humans have a fast path for facial recognition over many other mammals. Some in some people this is otherwise broken or has been co-opted to fast recognize something else.
- itisit 7mo ago
  > How were they able to achieve such perfect recreation from monkey Because the macaque study didn't decode faces from fMRI. They first used fMRI to locate the face patches, then used tungsten microelectrodes for single-unit electrophysiology to record spikes from individual face-selective neurons in ML/MF and AM. [0] Single-unit recordings capture individual spike patterns at a resolution fMRI, which averages across hundreds of thousands of neurons per voxel, simply cannot provide. [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC8088389 https://pmc.ncbi.nlm.nih.gov/articles/PMC8088389

62 points by katsee 7mo ago

Is there a case for non-dystopian applications for such a project, should it succeed? I get that we're all driven by curiosity, and the brain is very mysterious, but at some point I really wonder when scientists will start to taboo projects like this for ethical reasons, just like they currently taboo human cloning.

td2 7mo ago
I dont know anything about neurosience But if dreaming is related enouth to visual perception, maybe a dream recorder might be somewhat possible
- td2 7mo ago
  Edit, reading the github description, which i should have done sooner, hints at dreams.
  - bensyverson 7mo ago
    Every researcher working on this should be required to watch Until the End of the World.
katsee 7mo ago
Reconstruction of inner experience - imagery, maybe dreams & hallucinations, perceptual phenomena. Here is a recent study reconstructing the inner perception of optical illusions: https://www.science.org/doi/abs/10.1126/sciadv.adj3906 https://www.science.org/doi/abs/10.1126/sciadv.adj3906 Methods like MVPA (decoding among finite sets of cognitive state classes) are actually widely used for insight in cognitive neuroscience. Ethical concerns are discussed within the field; most papers had explicit ethics sections and discussions long before AI conferences required them for all submissions. In practice these experiments require a participant lying motionless (≈1-2 mm range) in an MRI scanner with controlled gaze and attention for many hours, and even then zero-shot reconstruction is not really possible; the SNR requires many repetitions.
MarcelOlsz 7mo ago
Not since Americans imported the Nazis - Von Braun et al.
aspenmartin 7mo ago
Dystopian applications are extremely impractical or impossible, this is a tool for neuroscience
- pixl97 7mo ago
  "This will never be used by bad guys" says the person immediately before their tools are used by bad guys.
  - aspenmartin 7mo ago
    No, it's "this tool cannot be used by bad guys or good guys, but can be used by highly funded labs that do neuroscience". It's something that freaks people out until they gradually learn what is actually involved https://ai.meta.com/blog/brain-ai-image-decoding-meg-magnetoencephalography/ https://ai.meta.com/blog/brain-ai-image-decoding-meg-magneto... [2023] https://ai.meta.com/blog/brain-ai-research-human-communication/ https://ai.meta.com/blog/brain-ai-research-human-communicati... [last year, focuses on decoding text] ^ There's a research team at Meta that studies this. You need an MEG -- thats $2-5M + the shielded room it lives in and the experts that can operate it. EEG doesn't work due to low spatial resolution and how finicky it is to place the electrodes to get a good signal The signals from neurons are just unbelievably tiny and are in an absolute sea of noisy trash. No one is ever going to read your thoughts without your consent (or by wrestling you into a big MEG, in which case you have bigger things to worry about). No one is going to be reading your dreams with any sort of accuracy either.
    - mandolingual 7mo ago
      And computers used to fill a room and require stacks of punch cards to use.
      - aspenmartin 7mo ago
        in both cases: physics decides what's possible

smusamashah 7mo ago

There was this research where faces were almost perfectly reconstructed from money's brain signals. How were they able to achieve such perfect recreation from monkey but not even close from human brain. https://www.bbc.co.uk/news/science-environment-40131242 https://www.bbc.co.uk/news/science-environment-40131242

pixl97 7mo ago
So, the question I have is, could the reassemble the data from a person who has face blindness? From my rather weak understanding on the subject, humans have a fast path for facial recognition over many other mammals. Some in some people this is otherwise broken or has been co-opted to fast recognize something else.
itisit 7mo ago
> How were they able to achieve such perfect recreation from monkey Because the macaque study didn't decode faces from fMRI. They first used fMRI to locate the face patches, then used tungsten microelectrodes for single-unit electrophysiology to record spikes from individual face-selective neurons in ML/MF and AM. [0] Single-unit recordings capture individual spike patterns at a resolution fMRI, which averages across hundreds of thousands of neurons per voxel, simply cannot provide. [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC8088389 https://pmc.ncbi.nlm.nih.gov/articles/PMC8088389