7 ms·

Really cool to see progress made here but this won't be available for public use any time soon (likely decades). One of the biggest challenges with decoding br

Really cool to see progress made here but this won't be available for public use any time soon (likely decades). One of the biggest challenges with decoding brain signals is getting a large number sensors that detect voltages from a very localized region of the brain. This study was done with ECoG (Eletro-Cortico-Gram) which involves implanting small electrodes directly on the surface of the brain. Nearly all consumer devices use EEG (Electro Encephelo Gram) which involves putting sensors on the surface of the skin. Commercially available ECoG is highly unlikely as it requires an extremely invasive brain surgery. For ethical reasons the implants from the study we're likely implanted to help diagnose existing life threatening medical issues. Decoding speech from EEG won't work as well as ECoG a number of reasons. First the physical distance between the sensors and the brain Means the signals you pick up aren’t localized. Second the skin and skull are great low pass filters and filter out really interesting signals at higher frequencies, 100-2KHZ. Additionally these signals have a really low signal power because they're correlated with neuronal spiking. ECoG does a really good job picking on these signals because the sensor is literally on the surface of the brain. Its really hard to pick up these signals reliably with EEG.

PetitPrince 7y ago
> Its really hard to pick up these signals reliably with EEG. If I may add : EEG is also a pain to put in place (you have to put gel, correctly put the cap etc.) and it's very easy to pollute your signal by merely moving a bit or even blinking.
HPsquared 7y ago
What's your opinion on Neuralink?
- akanet 7y ago
  Neuralink is even more speculative - they want to embed like a thousand tiny metal filaments deep into your brain.
  - 2008guy 7y ago
    How do you place electrodes without metal filaments? It’s been validated in monkeys.
    - mattkrause 7y ago
      Is that true? There was a throwaway comment about that at a press conference and lots of rumors (positive and negative), but the white paper only mentions “monkey” once—-and in a reference to another group’s paper.
      - 2008guy 7y ago
        Elon musk said on stage that they had put it in a monkey and had the monkey move a cursor with it. I’ve been following musk very closely since 2010 and he’s never blatantly lied so I think this will not be the first time. He has made errors about timelines and etc but never, ever blatantly lied about something so concrete. And it’s plausible to boot.
        erohead 7y ago
        > never blatantly lied. Funding secured
        morelisp 7y ago
        This is merely a mistatement of fact. Lying is what the poors do!
        2008guy 7y ago
        It wasn’t a lie...
        mattkrause 7y ago
        Not to swing my credentials around, but I actually record neural activity, from monkey brains, for a living. It can be unexpectedly hard in so many different ways. The dura covering the monkey brain is much tougher, the brain itself is larger, more convoluted, and moves more, even just from breathing and heartbeats). The animals have busy, clever little fingers, so the interface itself needs to be mechanically robust and durable because these implants need to last for years. I certainly want this to be true: with the exception of neuropixels, electrode technology has been depressingly stagnant. On the other hand, I need to see data before I get too excited and if I did have it, I'd be shouting it from the rooftops.
- bognition 7y ago
  I think they're a cool company and I even thought about applying. However the limiting factor here isn't the implant technology. Lots of groups have been building really great stuff for years (groups around the world have been recording from neurons in living animals for decades). The issue is there are ethical implications for performing such an invasive brain surgery on a healthy person. Right now the limit factor is finding suitable candidates. The ECoG implants are usually done to pinpoint where seizers start in the brain. The surgeons already have a good idea of the rough area from EEG but to zero in on the exact locus they need ECoG. So to find a candidate for a study you need someone who has epilepsy, their epilepsy needs to be bad enough to merit brain surgery, and the epileptic center cannot be the brain region you care about but it needs to be close enough to your target region that the same ECoG will cover both brain areas. So again I'm all for pushing this science forward. The more we learn about how the brain works the more we'll understand what makes us human. However, this isn't a technology problem right now. Its an ethical and medical one.
  - Ajedi32 7y ago
    > However, the limiting factor here isn't the implant technology. [...] The issue is there are ethical implications for performing such an invasive brain surgery on a healthy person. That sounds like a limitation of the implant technology to me. The reason there are ethical problems with performing invasive brain surgery on a healthy person is because the risks and downsides of currently available implant technology are significant. If getting a brain implant were as cheap, easy, and safe as getting a tattoo, the ethical problem would be largely solved.
    - jawns 7y ago
      > If getting a brain implant were as cheap, easy, and safe as getting a tattoo, the ethical problem would be largely solved. But how realistic is that? Your brain is a vital organ. It's encased in a hard skull. There is very little margin for error. It just doesn't strike me as the sort of procedure that could ever be made as cheap, easy, and safe as getting a tattoo -- at least not in our lifetime.
antupis 7y ago
Stupid question but could you do something very simple like on/off-switch with EEG and something similar than what they have done here?
- bognition 7y ago
  Not sure how that would change anything. Think of the brain as a crowd at a soccer field and to decode the brain you'd need a mic on each person to capture every conversation. ECoG is like having a mic shared between every 100 people. While EEG is like having a single mic that is 10K feet over the soccer stadium.
  - posterboy 7y ago
    This is my new favorite metaphor, if it's an analogy to how people sing, and how a person can sing somewhat randomly (free) yet controlled. The fact that there are two sides to the party chanting against each other, and the stadion speaker blurting over it occasionally, makes it so much better. Corallary, you need two microphones at least for simplicities sake, although the repetitivenes of the chants makes it a little easier. Most likely this will have good use in aphasia research.
- gridlockd 7y ago
  Yes, see EPOC
- garethsprice 7y ago
  Muse headset uses the "on/off" signal for meditation training. Has a fun stream API you can play with.
duckface 7y ago
I doubt this very much. Sparse signal reconstruction is a massive and very possible thing to do using IIRC various forms of FFT. I think this has already been done, and probably consumer devices will be using sparsity to reconstruct cortical signals with sufficient detail for this.
2008guy 7y ago
No, actually high density electrode implants are right around the corner. Watch the neuralink press event.
- bognition 7y ago
  The limiting factor here isn't implant technology. The limiting factor is the rate we can implant electrodes and do new science. Don't get me wrong, I love Elon as much as the next fanboy but there are serious ethical implications at play here that you can't just engineer your way around.
- empath75 7y ago
  > One engineering challenge is that the brain's chemical environment can cause many plastics to gradually deteriorate. Another challenge to chronic electrode implants is the inflammatory reaction to brain implants. Transmission of chemical messengers via neurons is impeded by a barrier-forming glial scar that occurs within weeks after insertion followed by progressive neurodegeneration, attenuating signal sensitivity. Furthermore, the thin electrodes which Neuralink uses, are more likely to break than thicker electrodes, and currently cannot be removed when broken or when rendered useless after glial scar forming. Yeah, you're not putting that in my brain. https://en.wikipedia.org/wiki/Neuralink https://en.wikipedia.org/wiki/Neuralink
  - 2008guy 7y ago
    The inflammation and damage is seen in traditional arrays that are large and rigid. A thread with low enough moment of inertia will likely not cause as much damage. And the damage is only important if you put the electrodes in an important place... we currently screw two giant lag screws into people’s heads and call it “deep brain stimulation” so I feel optimistic about the long game. But you’d still be right to be wary. And if I had it my way, this technology would never be allowed to exist at all...
    - Balgair 7y ago
      > A thread with low enough moment of inertia will likely not cause as much damage. Shear forces cause glial scarring?
king07828 7y ago
Has there been any work to use a neural network to generate/simulate ECoG signal output from an EEG signal input? (My Google-fu only gives definitions and distinctions for ECoG and EEG). Almost sounds similar in concept to deep learning super sampling (DLSS), i.e., taking a low resolution image/signal (EEG) and using a neural network to generate/simulate a high resolution image/signal (ECoG).
- raverbashing 7y ago
  The problem sounds similar but it's completely different DLSS kinda works because we "know" what is what thing in a photo. EEG to ECOG would be trying to figure out a painting (that could be anything, from any painter) from a significant distance behind a frosted glass
  - king07828 7y ago
    > DLSS kinda works because we "know" what is what thing in a photo. I thought the reason DLSS works is because the same rendering algorithm is used to generate the low resolution image and the high resolution image and the neural network merely learns a filter between the two. Take a patient with ECoG implant(s), put EEG sensors on the patient, and hit record. You now have the same rendering mechanism (the brain) generating a low resolution signal (EEG) and a high resolution signal (ECoG). However, back to DLSS, if the low resolution signal is a single pixel, then generating a 4k image from just that single pixel may not be very fruitful. Still, it would be interesting to see an attempt at using a generative adversarial network (GAN) to generate an ECoG from an EEG. And if it doesn't work, then make a determination of how much more EEG sensitivity is needed before it will work.
hughw 7y ago
Seismologist here. The problems you describe sound reminiscent of those in seismic imaging. Ideally you'd bury geophones below the weathered layer (typically 3 m) at the surface, to get ideal coupling. In practice that's not economic at scale, and so you plant them on the surface, and account for the non-linear wave transmission through the weathered layer by clever math and by collecting more samples. There's been a forty-year evolution in these techniques. The cheap, noisy technique might prevail if scientists keep refining the craft by tiny improvements.
- stanfordkid 7y ago
  I looked into the problems in this field in the past, my impression is that the difficulty lies in how the signal diffuses through the skull and top-most layers of the brain. It’s all about resolution, and this sounds like a very similar problem. The question always is how much non-linearity is there, and what’s the frequency and density of the sensors relative to how often you have to sample to characterize it. Very excited to see how the field develops!
- kensai 7y ago
  Neurologist here. There is an inherent problem of physics. The skull acts as a low-pass filter of the brain's activity. You lose a lot of information if you don't go under the skull. I don't think EEGs can really give the spatial and temporal resolution we really need to extract the necessary information for thought decoding (or encoding).
  - youareostriches 7y ago
    I’m curious of your opinion on Openwater: https://www.openwater.cc/technology https://www.openwater.cc/technology
  - dnautics 7y ago
    you don't need to decode thought, you just need to decode information at some really low bitrate to export information (say text). Working with a NN, the brain can probably negotiate the relevant parameters (let's say frequency modulation across one channels with four bins, across 4 spots on the brain), that's 4^4 = 64 values across whatever timebox of resolution you get. That's enough to encode english letters, which itself probably is an underprovisioned mechanism of data transfer).
lioeters 7y ago
I've been fascinated by this topic since a couple decades ago, when I participated in a research study involving EEG and word recognition. Progress seems to be slow but steady, with applications getting more accurate and practical. Your point about invasive implants being impractical for commercial use made me wonder.. I searched the first phrase that popped into my head, "Non-invasive Brain-Computer Interface". Looks like there's promising research on significantly improving sensitivity/resolution of EEG singals. - First Ever Non-invasive Brain-Computer Interface Developed - https://www.technologynetworks.com/informatics/news/first-ever-non-invasive-brain-computer-interface-developed-320941 https://www.technologynetworks.com/informatics/news/first-ev... (2019) - Noninvasive neuroimaging enhances continuous neural tracking for robotic device control - https://robotics.sciencemag.org/content/4/31/eaaw6844 https://robotics.sciencemag.org/content/4/31/eaaw6844 Still, your prediction of "likely decades" sounds realistic. I'm hoping for an affordable, non-invasive brain-computer interface to be as widely used as the keyboard, mouse, or microphone.
- dnautics 7y ago
  I don't think this is the first ever non-invasive bci. I recall decades ago there was a BCI where you could control between left and right literally by activating the appropriate hemisphere and triggering a relative potentiometer. Extremely crude, but it worked. IIRC the instructions were "to go left, think 'go left'"; "to go right, relax". After some number of sessions (I think I remember it being "a lot") the user's brain would automatically do the correct thing.
turingbike 7y ago
There was a great article on the front page yesterday, When to Assume Neural Networks Can Solve a Problem, https://news.ycombinator.com/item?id=22717367 https://news.ycombinator.com/item?id=22717367 . Case 2 is very relevant here, basically: if you have solved the problem with access to lots of data, usually you can adapt to a lower-data regime. I am way out of my element talking about brain surgery and sensors. However, one thing that I do well is say "you shouldn't bet against neural networks", which is a great way to be right on a few-year time horizon.

PetitPrince 7y ago

> Its really hard to pick up these signals reliably with EEG. If I may add : EEG is also a pain to put in place (you have to put gel, correctly put the cap etc.) and it's very easy to pollute your signal by merely moving a bit or even blinking.

HPsquared 7y ago

What's your opinion on Neuralink?

akanet 7y ago
Neuralink is even more speculative - they want to embed like a thousand tiny metal filaments deep into your brain.
- 2008guy 7y ago
  How do you place electrodes without metal filaments? It’s been validated in monkeys.
  - mattkrause 7y ago
    Is that true? There was a throwaway comment about that at a press conference and lots of rumors (positive and negative), but the white paper only mentions “monkey” once—-and in a reference to another group’s paper.
    - 2008guy 7y ago
      Elon musk said on stage that they had put it in a monkey and had the monkey move a cursor with it. I’ve been following musk very closely since 2010 and he’s never blatantly lied so I think this will not be the first time. He has made errors about timelines and etc but never, ever blatantly lied about something so concrete. And it’s plausible to boot.
      - erohead 7y ago
        > never blatantly lied. Funding secured
        morelisp 7y ago
        This is merely a mistatement of fact. Lying is what the poors do!
        2008guy 7y ago
        It wasn’t a lie...
      - mattkrause 7y ago
        Not to swing my credentials around, but I actually record neural activity, from monkey brains, for a living. It can be unexpectedly hard in so many different ways. The dura covering the monkey brain is much tougher, the brain itself is larger, more convoluted, and moves more, even just from breathing and heartbeats). The animals have busy, clever little fingers, so the interface itself needs to be mechanically robust and durable because these implants need to last for years. I certainly want this to be true: with the exception of neuropixels, electrode technology has been depressingly stagnant. On the other hand, I need to see data before I get too excited and if I did have it, I'd be shouting it from the rooftops.
bognition 7y ago
I think they're a cool company and I even thought about applying. However the limiting factor here isn't the implant technology. Lots of groups have been building really great stuff for years (groups around the world have been recording from neurons in living animals for decades). The issue is there are ethical implications for performing such an invasive brain surgery on a healthy person. Right now the limit factor is finding suitable candidates. The ECoG implants are usually done to pinpoint where seizers start in the brain. The surgeons already have a good idea of the rough area from EEG but to zero in on the exact locus they need ECoG. So to find a candidate for a study you need someone who has epilepsy, their epilepsy needs to be bad enough to merit brain surgery, and the epileptic center cannot be the brain region you care about but it needs to be close enough to your target region that the same ECoG will cover both brain areas. So again I'm all for pushing this science forward. The more we learn about how the brain works the more we'll understand what makes us human. However, this isn't a technology problem right now. Its an ethical and medical one.
- Ajedi32 7y ago
  > However, the limiting factor here isn't the implant technology. [...] The issue is there are ethical implications for performing such an invasive brain surgery on a healthy person. That sounds like a limitation of the implant technology to me. The reason there are ethical problems with performing invasive brain surgery on a healthy person is because the risks and downsides of currently available implant technology are significant. If getting a brain implant were as cheap, easy, and safe as getting a tattoo, the ethical problem would be largely solved.
  - jawns 7y ago
    > If getting a brain implant were as cheap, easy, and safe as getting a tattoo, the ethical problem would be largely solved. But how realistic is that? Your brain is a vital organ. It's encased in a hard skull. There is very little margin for error. It just doesn't strike me as the sort of procedure that could ever be made as cheap, easy, and safe as getting a tattoo -- at least not in our lifetime.

antupis 7y ago

Stupid question but could you do something very simple like on/off-switch with EEG and something similar than what they have done here?

bognition 7y ago
Not sure how that would change anything. Think of the brain as a crowd at a soccer field and to decode the brain you'd need a mic on each person to capture every conversation. ECoG is like having a mic shared between every 100 people. While EEG is like having a single mic that is 10K feet over the soccer stadium.
- posterboy 7y ago
  This is my new favorite metaphor, if it's an analogy to how people sing, and how a person can sing somewhat randomly (free) yet controlled. The fact that there are two sides to the party chanting against each other, and the stadion speaker blurting over it occasionally, makes it so much better. Corallary, you need two microphones at least for simplicities sake, although the repetitivenes of the chants makes it a little easier. Most likely this will have good use in aphasia research.
gridlockd 7y ago
Yes, see EPOC
garethsprice 7y ago
Muse headset uses the "on/off" signal for meditation training. Has a fun stream API you can play with.

duckface 7y ago

I doubt this very much. Sparse signal reconstruction is a massive and very possible thing to do using IIRC various forms of FFT. I think this has already been done, and probably consumer devices will be using sparsity to reconstruct cortical signals with sufficient detail for this.

2008guy 7y ago

No, actually high density electrode implants are right around the corner. Watch the neuralink press event.

bognition 7y ago
The limiting factor here isn't implant technology. The limiting factor is the rate we can implant electrodes and do new science. Don't get me wrong, I love Elon as much as the next fanboy but there are serious ethical implications at play here that you can't just engineer your way around.
empath75 7y ago
> One engineering challenge is that the brain's chemical environment can cause many plastics to gradually deteriorate. Another challenge to chronic electrode implants is the inflammatory reaction to brain implants. Transmission of chemical messengers via neurons is impeded by a barrier-forming glial scar that occurs within weeks after insertion followed by progressive neurodegeneration, attenuating signal sensitivity. Furthermore, the thin electrodes which Neuralink uses, are more likely to break than thicker electrodes, and currently cannot be removed when broken or when rendered useless after glial scar forming. Yeah, you're not putting that in my brain. https://en.wikipedia.org/wiki/Neuralink https://en.wikipedia.org/wiki/Neuralink
- 2008guy 7y ago
  The inflammation and damage is seen in traditional arrays that are large and rigid. A thread with low enough moment of inertia will likely not cause as much damage. And the damage is only important if you put the electrodes in an important place... we currently screw two giant lag screws into people’s heads and call it “deep brain stimulation” so I feel optimistic about the long game. But you’d still be right to be wary. And if I had it my way, this technology would never be allowed to exist at all...
  - Balgair 7y ago
    > A thread with low enough moment of inertia will likely not cause as much damage. Shear forces cause glial scarring?

king07828 7y ago

Has there been any work to use a neural network to generate/simulate ECoG signal output from an EEG signal input? (My Google-fu only gives definitions and distinctions for ECoG and EEG). Almost sounds similar in concept to deep learning super sampling (DLSS), i.e., taking a low resolution image/signal (EEG) and using a neural network to generate/simulate a high resolution image/signal (ECoG).

raverbashing 7y ago
The problem sounds similar but it's completely different DLSS kinda works because we "know" what is what thing in a photo. EEG to ECOG would be trying to figure out a painting (that could be anything, from any painter) from a significant distance behind a frosted glass
- king07828 7y ago
  > DLSS kinda works because we "know" what is what thing in a photo. I thought the reason DLSS works is because the same rendering algorithm is used to generate the low resolution image and the high resolution image and the neural network merely learns a filter between the two. Take a patient with ECoG implant(s), put EEG sensors on the patient, and hit record. You now have the same rendering mechanism (the brain) generating a low resolution signal (EEG) and a high resolution signal (ECoG). However, back to DLSS, if the low resolution signal is a single pixel, then generating a 4k image from just that single pixel may not be very fruitful. Still, it would be interesting to see an attempt at using a generative adversarial network (GAN) to generate an ECoG from an EEG. And if it doesn't work, then make a determination of how much more EEG sensitivity is needed before it will work.

hughw 7y ago

Seismologist here. The problems you describe sound reminiscent of those in seismic imaging. Ideally you'd bury geophones below the weathered layer (typically 3 m) at the surface, to get ideal coupling. In practice that's not economic at scale, and so you plant them on the surface, and account for the non-linear wave transmission through the weathered layer by clever math and by collecting more samples. There's been a forty-year evolution in these techniques. The cheap, noisy technique might prevail if scientists keep refining the craft by tiny improvements.

stanfordkid 7y ago
I looked into the problems in this field in the past, my impression is that the difficulty lies in how the signal diffuses through the skull and top-most layers of the brain. It’s all about resolution, and this sounds like a very similar problem. The question always is how much non-linearity is there, and what’s the frequency and density of the sensors relative to how often you have to sample to characterize it. Very excited to see how the field develops!
kensai 7y ago
Neurologist here. There is an inherent problem of physics. The skull acts as a low-pass filter of the brain's activity. You lose a lot of information if you don't go under the skull. I don't think EEGs can really give the spatial and temporal resolution we really need to extract the necessary information for thought decoding (or encoding).
- youareostriches 7y ago
  I’m curious of your opinion on Openwater: https://www.openwater.cc/technology https://www.openwater.cc/technology
- dnautics 7y ago
  you don't need to decode thought, you just need to decode information at some really low bitrate to export information (say text). Working with a NN, the brain can probably negotiate the relevant parameters (let's say frequency modulation across one channels with four bins, across 4 spots on the brain), that's 4^4 = 64 values across whatever timebox of resolution you get. That's enough to encode english letters, which itself probably is an underprovisioned mechanism of data transfer).

lioeters 7y ago

I've been fascinated by this topic since a couple decades ago, when I participated in a research study involving EEG and word recognition. Progress seems to be slow but steady, with applications getting more accurate and practical. Your point about invasive implants being impractical for commercial use made me wonder.. I searched the first phrase that popped into my head, "Non-invasive Brain-Computer Interface". Looks like there's promising research on significantly improving sensitivity/resolution of EEG singals. - First Ever Non-invasive Brain-Computer Interface Developed - https://www.technologynetworks.com/informatics/news/first-ever-non-invasive-brain-computer-interface-developed-320941 https://www.technologynetworks.com/informatics/news/first-ev... (2019) - Noninvasive neuroimaging enhances continuous neural tracking for robotic device control - https://robotics.sciencemag.org/content/4/31/eaaw6844 https://robotics.sciencemag.org/content/4/31/eaaw6844 Still, your prediction of "likely decades" sounds realistic. I'm hoping for an affordable, non-invasive brain-computer interface to be as widely used as the keyboard, mouse, or microphone.

dnautics 7y ago
I don't think this is the first ever non-invasive bci. I recall decades ago there was a BCI where you could control between left and right literally by activating the appropriate hemisphere and triggering a relative potentiometer. Extremely crude, but it worked. IIRC the instructions were "to go left, think 'go left'"; "to go right, relax". After some number of sessions (I think I remember it being "a lot") the user's brain would automatically do the correct thing.

turingbike 7y ago

There was a great article on the front page yesterday, When to Assume Neural Networks Can Solve a Problem, https://news.ycombinator.com/item?id=22717367 https://news.ycombinator.com/item?id=22717367 . Case 2 is very relevant here, basically: if you have solved the problem with access to lots of data, usually you can adapt to a lower-data regime. I am way out of my element talking about brain surgery and sensors. However, one thing that I do well is say "you shouldn't bet against neural networks", which is a great way to be right on a few-year time horizon.