5 ms·

All the talk of implants makes me wonder what's possible without a direct physical link to the brain. Like using an EEG to measure brain signals and transform t

by bwood 6y ago

All the talk of implants makes me wonder what's possible without a direct physical link to the brain. Like using an EEG to measure brain signals and transform them into instructions, images, etc. Here's some research where they use the EEG to train a model that reconstructs what people are seeing in their visual cortex [0] All the reconstructions are super low resolution, but I wonder how much better they can get? I'd be much more open to brain-computer interfaces if it didn't involve drilling through my skull. [0] https://scitechdaily.com/image-reconstruction-from-human-brain-waves-in-real-time-video/ https://scitechdaily.com/image-reconstruction-from-human-bra...

malwarebytess 6y ago
Tangentially related, but I wanted to post an article and a paper. I remember hearing way back around 2010 that Valve was messing around with BCI's. Valve has as recently as 2019 talked about EEG's and gaming [0]. Also recall reading about similar concepts over a decade ago. [1] [0] https://venturebeat.com/2019/03/24/valve-psychologist-explores-controlling-games-directly-with-your-brain/ https://venturebeat.com/2019/03/24/valve-psychologist-explor... [1] https://vpa.sabanciuniv.edu/phpBB2/vpa_views.php?s=4&serial=29 https://vpa.sabanciuniv.edu/phpBB2/vpa_views.php?s=4&serial=...
bognition 6y ago
Most if not all of the very interesting signals are in the 100hz - 10Khz range. These signals are also very low voltage so. Add in that the skulls acts as a low pass filter and you end up in a situation where you just will not be able to recover the information needed to decode what the brain is doing at the required resolution. You can use ECoG which is like EEG but placed on under dura-mater (a protective bag around the brain) but this is also highly invasive. Also the spatial resolution is pretty bad. You can get a hundred or so sensors in a few square centimeters.
- PH01 6y ago
  What signals do you mean? Brain waves measured in EEG typically range from 0.5 Hz and above.
  - bognition 6y ago
    The signals you pick up between 0.5 Hz and 40Hz are mostly likely "population" level signals or put another way the average activity of millions of neurons spread across large swaths of the brain. A good analogy re: trying to decode the brain from EEG would be akin to trying to decode what is going on in a soccer match but only using the noise of the crowds.
    - PH01 6y ago
      How are signals up to 10Khz generated? I thought the absolute refractory period of a neuronal action potential limits the rate at which they can fire.
      - mattkrause 6y ago
        From start to finish, an individual action potential lasts about 1ms, followed by a few millisecond refractory period. There is probably not much biologically-relevant activity happening at faster timescales. However, one often wants to "sort" spikes so that those arising from individual neurons are grouped together. To do so, you need to sample fast enough to capture the shape of the action potential, not just the fact that it happened. This usually requires sampling rates in the 10s of kHz. (~30 kHz would be good).
        PH01 6y ago
        So, to be clear, the signal of interest does not go above 1kHz. Obviously if you want to sample a signal you have to use a higher frqeuency.
        mattkrause 6y ago
        Eh, it depends on what you're calling "the signal of interest." A spike train, the output of a neuron, can reasonably be represented as a binary signal (1=spike, 0=no spike) at 1 kHz. In fact, I'd say this is nearly standard. The neuron's membrane potential, a combination of its current inputs, intrinsic properties, and recent history, changes on faster timescales. Whatever processing the brain does probably does not use this information itself, but one probably wants access to it anyway for experimental and practical reasons. The signal that an extracellular electrode picks up isn't that 1 kHz spike train. It's many of them mixed together and you'll need finer features to tell them apart. In other words, you can't just invoke Nyquist, sample at 2 kHz, and call it a day.
yters 6y ago
I have an OpenBCI headset that I have been messing with. Signal is definitely noisy, but they've been able to get a degree of mind control working, like flying little helicopters.
PH01 6y ago
Doesn't reconstruction like this rely on the decoding mechanism being trained on the original images? This doesn't scale to reconstructing any visual image, only to differentiating between those the decoder has already seen. It doubt very much that it reconstructs from the visual cortext. That is a non-trivial task with fMRI which has a much higher SNR than EEG.
- mattkrause 6y ago
  Here's their preprint. It's basically using a pretrained VGG network as a latent space, which lets them synthesize images. https://www.biorxiv.org/content/10.1101/787101v3 https://www.biorxiv.org/content/10.1101/787101v3

by bwood 6y ago

malwarebytess 6y ago

Tangentially related, but I wanted to post an article and a paper. I remember hearing way back around 2010 that Valve was messing around with BCI's. Valve has as recently as 2019 talked about EEG's and gaming [0]. Also recall reading about similar concepts over a decade ago. [1] [0] https://venturebeat.com/2019/03/24/valve-psychologist-explores-controlling-games-directly-with-your-brain/ https://venturebeat.com/2019/03/24/valve-psychologist-explor... [1] https://vpa.sabanciuniv.edu/phpBB2/vpa_views.php?s=4&serial=29 https://vpa.sabanciuniv.edu/phpBB2/vpa_views.php?s=4&serial=...

bognition 6y ago

Most if not all of the very interesting signals are in the 100hz - 10Khz range. These signals are also very low voltage so. Add in that the skulls acts as a low pass filter and you end up in a situation where you just will not be able to recover the information needed to decode what the brain is doing at the required resolution. You can use ECoG which is like EEG but placed on under dura-mater (a protective bag around the brain) but this is also highly invasive. Also the spatial resolution is pretty bad. You can get a hundred or so sensors in a few square centimeters.

PH01 6y ago
What signals do you mean? Brain waves measured in EEG typically range from 0.5 Hz and above.
- bognition 6y ago
  The signals you pick up between 0.5 Hz and 40Hz are mostly likely "population" level signals or put another way the average activity of millions of neurons spread across large swaths of the brain. A good analogy re: trying to decode the brain from EEG would be akin to trying to decode what is going on in a soccer match but only using the noise of the crowds.
  - PH01 6y ago
    How are signals up to 10Khz generated? I thought the absolute refractory period of a neuronal action potential limits the rate at which they can fire.
    - mattkrause 6y ago
      From start to finish, an individual action potential lasts about 1ms, followed by a few millisecond refractory period. There is probably not much biologically-relevant activity happening at faster timescales. However, one often wants to "sort" spikes so that those arising from individual neurons are grouped together. To do so, you need to sample fast enough to capture the shape of the action potential, not just the fact that it happened. This usually requires sampling rates in the 10s of kHz. (~30 kHz would be good).
      - PH01 6y ago
        So, to be clear, the signal of interest does not go above 1kHz. Obviously if you want to sample a signal you have to use a higher frqeuency.
        mattkrause 6y ago
        Eh, it depends on what you're calling "the signal of interest." A spike train, the output of a neuron, can reasonably be represented as a binary signal (1=spike, 0=no spike) at 1 kHz. In fact, I'd say this is nearly standard. The neuron's membrane potential, a combination of its current inputs, intrinsic properties, and recent history, changes on faster timescales. Whatever processing the brain does probably does not use this information itself, but one probably wants access to it anyway for experimental and practical reasons. The signal that an extracellular electrode picks up isn't that 1 kHz spike train. It's many of them mixed together and you'll need finer features to tell them apart. In other words, you can't just invoke Nyquist, sample at 2 kHz, and call it a day.

yters 6y ago

I have an OpenBCI headset that I have been messing with. Signal is definitely noisy, but they've been able to get a degree of mind control working, like flying little helicopters.

PH01 6y ago

Doesn't reconstruction like this rely on the decoding mechanism being trained on the original images? This doesn't scale to reconstructing any visual image, only to differentiating between those the decoder has already seen. It doubt very much that it reconstructs from the visual cortext. That is a non-trivial task with fMRI which has a much higher SNR than EEG.

mattkrause 6y ago
Here's their preprint. It's basically using a pretrained VGG network as a latent space, which lets them synthesize images. https://www.biorxiv.org/content/10.1101/787101v3 https://www.biorxiv.org/content/10.1101/787101v3