5 ms·

Is this really sound criticism? The first iterations of lots of technology were worse than their contemporaries. You could easily out-ride the first car with a

Is this really sound criticism? The first iterations of lots of technology were worse than their contemporaries. You could easily out-ride the first car with a horse in both speed an distance. You could out-calculate the first computers. The point is the potential the technology has. Here they are translating actual brain patterns to words. That's pretty awesome.

AussieWog93 5y ago
I should point that I studied a PhD in the application of Machine Learning to Neurosignal decoding back in 2017-2018 (dropped out after one year), so this isn't just criticism I've plucked out the air. I spent a solid 6 months intimately learning the pros and cons of various existing BCI systems, as well as the exact methods researchers use to make their technology look like a breakthrough when it isn't. Specifically, here, they've created a system that can decode one of 50 symbols - a mere 1 bit more than the English alphabet - and described the symbols as "words" so that the reader thinks each symbol carries much more information than it actually does. They've also cherry-picked the patient that responded the best. When you peel back the hype, this is about the same performance we've been getting since '08 for an invasive, subdural, non-penetrative array.
- thaw13579 5y ago
  Any chance you can please share a reference for that earlier comparable work? Not challenging your view, just curious
  - AussieWog93 5y ago
    Oh God, I started looking up the papers and my brain started melting. I'd genuinely forgotten about SSVEP. Anyway, knock yourself out with research: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_yhi=2010&q=motor+imagery+bci+ecog+ITR&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_yhi... Here's one from 2006 that got 35bpm without an implant (the one in the article would be closer to 60, but that's to be expected as it's invasive): https://d1wqtxts1xzle7.cloudfront.net/46069183/the_berlin_brain_computer_interface_eeg_based_communication.pdf?1464613871=&response-content-disposition=inline%3B+filename%3DThe_Berlin_Brain_Computer_Interface_EEG.pdf&Expires=1626675385&Signature=DIL7lmRh0uGvshGmHmekrNSKLXoyLeA~rBow80kC935JIhjuL2bPX9HF3fr8ilPIdBYcgDKuEGiz~H43niDmwVkxRHIE9Q~Ydwzn0C~e39C43771PNeWH-MB7l6WdMayixLeWSwQxF7PXu7psZLwUUvs6KpapTdWxfDiED0oBLiz-igPLRgFssm5mRpCKOvyi3BwjbOWa-fhsM~O2Qn63gM7V6jhZSKdFlNP7ya5Rj0RYcS3OcjUqwePOSi9leGBFk48wJT1YrXGvWu~Y9v8Ztxr7qONEHfIEyE~xP1wmYGnWFCNS-1HUG4ZoCm5Y3oy4d8gTJK9yNJQSmW4~DX2DA__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA https://d1wqtxts1xzle7.cloudfront.net/46069183/the_berlin_br... Here's another from 2010 with a similar result under similar conditions: https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pdf https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pd...
    - swordsmith 5y ago
      > Here's another from 2010 with a similar result under similar conditions: https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pd https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pd... Within the abstract: > The average accuracy and information transfer rate are 92.25% and 37.62 bits per minute, which is translated in the speller with an average speed of 5.51 letters per minute. 5.51 letters per minute, not words. The work you cited is not comparable to the UCSF work at all. It seems you are measuring performance in terms of bit-rate (i.e. based on how many symbols per minute), which makes sense when you are using a cursor-based speller. This approach is not a correct measurement of bitrate with this speech-motor decoder, however, as words are being decoded based on the syllables contained within it. The decoding model is trained to recognize 50 specific combinations of syllables, and the total number of unique single syllable phonemes is about 44.
      - AussieWog93 5y ago
        >5.51 letters per minute, not words Again, it is misleading at best to user a measure of "words per second" when you're restricted to a set of 50 of them. A keyboard that had both English and Cyrillic characters in it would have 59 unique symbols. >It seems you are measuring performance in terms of bit-rate (i.e. based on how many symbols per minute), which makes sense when you are using a cursor-based speller. I genuinely fail to see what the cursor has to do with anything. Communication speed is communication speed. >This approach is not a correct measurement of bitrate with this speech-motor decoder, however, as words are being decoded based on the syllables contained within it. Just as symbols in cursor tasks are decoded based on relative position?
- a-dub 5y ago
  krishna shenoy has some nice papers that apply information theory to bmis (albeit fused to a T9 style text entry task). either way, they attempt to quantify, in bits, the bitrate of the bmi. (which was, back then, quite low... a few bps) i think it gets rather muddy as there isn't really a good metric for raw signal quality (afaik). there's cell tuning and number of spiking channels, but still not a great measure of snr for bmi work (afaik). often times people will apply measures to the outputs of their systems, like task performance, but part of the problem there is that often the state model has varying quality and suitability to task, so it can be difficult to disambiguate signal quality from state model performance. (of course, in speech recognition they don't care, the game is to minimize WER and maximize decoding speed and whether language or acoustics (at least when they were separate) get you there, it doesn't matter)
  - AussieWog93 5y ago
    >i think it gets rather muddy as there isn't really a good metric for raw signal quality (afaik). there's cell tuning and number of spiking channels, but still not a great measure of snr for bmi work (afaik). often times people will apply measures to the outputs of their systems, like task performance, but part of the problem there is that often the state model has varying quality and suitability to task, so it can be difficult to disambiguate signal quality from state model performance. This was something I picked up on back in 2017. I did manage to come up with a definition of SNR that made some sense (basically the euclidean distance between symbol menas, divided by the noise level along the vector connecting the two symbols, assuming the feature space was basically an N-dimensional QAM signal using features 1...N instead of amplitude and phase) - but even then that didn't take into account the fact that the noise was neither well-approximated by AWGN biased nor even constant... And of course, as you said, you could get a bad SNR just because you're extracting the wrong features (although, to be fair, the same problem can exist in telecoms too).
    - a-dub 5y ago
      hah, it's funny. i come at all of this from a sensorimotor control view (even though i was very interested in speech and language- as for a cs person discrete stuff is easier to reason about) and while i can appreciate ideas like this and the shenoy lab stuff, when i worked on this stuff in practice we had no discrete symbols as we decoded continuous variables like positions, velocities, angles and torques- which didn't seem to have clean mappings into comms/noisy channel theory/info theory.
- swordsmith 5y ago
  From your top level comment: > a nurse a laminated piece of A4 paper and a patient who can blink get about 15 characters How is this a relevant comparison? 15 characters per minute is much less than the 15 WORDS per minute performance this work demonstrates > I spent a solid 6 months intimately learning the pros and cons of various existing BCI systems, as well as the exact methods researchers use to make their technology look like a breakthrough when it isn't. Then you should know this is a huge deal. > Specifically, here, they've created a system that can decode one of 50 symbols Previous motor and speech neuroprosthetic systems focused on pointing (controlling cursor) and more recently, handwriting (https://www.nature.com/articles/s41586-021-03506-2 https://www.nature.com/articles/s41586-021-03506-2). This work goes gives communication rate similar to the handwriting work, but decoding speech from the motor cortex has been much less understood than that of simple motor movements such as cursor position and velocity control. Even more impressive, the test subject is not even a native English speaker. > They've also cherry-picked the patient that responded the best. Bravo-1 is the only subject they had. > When you peel back the hype, this is about the same performance we've been getting since '08 for an invasive, subdural, non-penetrative array. This is completely, utterly false. See (https://stacks.stanford.edu/file/druid:jx921pv3255/TechnicalReport01_GPT_BCIs-2021-05-20.pdf https://stacks.stanford.edu/file/druid:jx921pv3255/Technical...) for survey of performance of typing BCI. You seem to love to cite your background as a BCI PhD dropout. I should also point out that I have a completed PhD in invasive neuroprosthetics and still work in the field (not that matters when anyone can look up the sources and judge for themselves).
  - AussieWog93 5y ago
    >How is this a relevant comparison? 15 characters per minute is much less than the 15 WORDS per minute performance this work demonstrates 15 words from a set of 50 (6 bits per "word" vs. 5 per letter in the English alphabet). It's like saying a 100 baud telegraph machine can decode 300 words per second, just so long as those words come from the set of "dot" and "dash". >but decoding speech from the motor cortex has been much less understood than that of simple motor movements such as cursor position and velocity control. Cursor position and velocity are outputs, the input is still a self-paced motor imagery task. >Bravo-1 is the only subject they had. Fair cop. Maybe they'll get genuinely impressive results with their next patient. >This is completely, utterly false. See (https://stacks.stanford.edu/file/druid:jx921pv3255/Technical https://stacks.stanford.edu/file/druid:jx921pv3255/Technical...) for survey of performance of typing BCI. I posted a link in a comment below showing an ITR of 35bpm from scalp EEG from pre-2010. >I have a completed PhD in invasive neuroprosthetics and still work in the field I am truly sorry for your loss.

AussieWog93 5y ago

I should point that I studied a PhD in the application of Machine Learning to Neurosignal decoding back in 2017-2018 (dropped out after one year), so this isn't just criticism I've plucked out the air. I spent a solid 6 months intimately learning the pros and cons of various existing BCI systems, as well as the exact methods researchers use to make their technology look like a breakthrough when it isn't. Specifically, here, they've created a system that can decode one of 50 symbols - a mere 1 bit more than the English alphabet - and described the symbols as "words" so that the reader thinks each symbol carries much more information than it actually does. They've also cherry-picked the patient that responded the best. When you peel back the hype, this is about the same performance we've been getting since '08 for an invasive, subdural, non-penetrative array.

thaw13579 5y ago
Any chance you can please share a reference for that earlier comparable work? Not challenging your view, just curious
- AussieWog93 5y ago
  Oh God, I started looking up the papers and my brain started melting. I'd genuinely forgotten about SSVEP. Anyway, knock yourself out with research: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_yhi=2010&q=motor+imagery+bci+ecog+ITR&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_yhi... Here's one from 2006 that got 35bpm without an implant (the one in the article would be closer to 60, but that's to be expected as it's invasive): https://d1wqtxts1xzle7.cloudfront.net/46069183/the_berlin_brain_computer_interface_eeg_based_communication.pdf?1464613871=&response-content-disposition=inline%3B+filename%3DThe_Berlin_Brain_Computer_Interface_EEG.pdf&Expires=1626675385&Signature=DIL7lmRh0uGvshGmHmekrNSKLXoyLeA~rBow80kC935JIhjuL2bPX9HF3fr8ilPIdBYcgDKuEGiz~H43niDmwVkxRHIE9Q~Ydwzn0C~e39C43771PNeWH-MB7l6WdMayixLeWSwQxF7PXu7psZLwUUvs6KpapTdWxfDiED0oBLiz-igPLRgFssm5mRpCKOvyi3BwjbOWa-fhsM~O2Qn63gM7V6jhZSKdFlNP7ya5Rj0RYcS3OcjUqwePOSi9leGBFk48wJT1YrXGvWu~Y9v8Ztxr7qONEHfIEyE~xP1wmYGnWFCNS-1HUG4ZoCm5Y3oy4d8gTJK9yNJQSmW4~DX2DA__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA https://d1wqtxts1xzle7.cloudfront.net/46069183/the_berlin_br... Here's another from 2010 with a similar result under similar conditions: https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pdf https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pd...
  - swordsmith 5y ago
    > Here's another from 2010 with a similar result under similar conditions: https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pd https://pure.ulster.ac.uk/ws/files/11410334/cecotti_tnsre.pd... Within the abstract: > The average accuracy and information transfer rate are 92.25% and 37.62 bits per minute, which is translated in the speller with an average speed of 5.51 letters per minute. 5.51 letters per minute, not words. The work you cited is not comparable to the UCSF work at all. It seems you are measuring performance in terms of bit-rate (i.e. based on how many symbols per minute), which makes sense when you are using a cursor-based speller. This approach is not a correct measurement of bitrate with this speech-motor decoder, however, as words are being decoded based on the syllables contained within it. The decoding model is trained to recognize 50 specific combinations of syllables, and the total number of unique single syllable phonemes is about 44.
    - AussieWog93 5y ago
      >5.51 letters per minute, not words Again, it is misleading at best to user a measure of "words per second" when you're restricted to a set of 50 of them. A keyboard that had both English and Cyrillic characters in it would have 59 unique symbols. >It seems you are measuring performance in terms of bit-rate (i.e. based on how many symbols per minute), which makes sense when you are using a cursor-based speller. I genuinely fail to see what the cursor has to do with anything. Communication speed is communication speed. >This approach is not a correct measurement of bitrate with this speech-motor decoder, however, as words are being decoded based on the syllables contained within it. Just as symbols in cursor tasks are decoded based on relative position?
a-dub 5y ago
krishna shenoy has some nice papers that apply information theory to bmis (albeit fused to a T9 style text entry task). either way, they attempt to quantify, in bits, the bitrate of the bmi. (which was, back then, quite low... a few bps) i think it gets rather muddy as there isn't really a good metric for raw signal quality (afaik). there's cell tuning and number of spiking channels, but still not a great measure of snr for bmi work (afaik). often times people will apply measures to the outputs of their systems, like task performance, but part of the problem there is that often the state model has varying quality and suitability to task, so it can be difficult to disambiguate signal quality from state model performance. (of course, in speech recognition they don't care, the game is to minimize WER and maximize decoding speed and whether language or acoustics (at least when they were separate) get you there, it doesn't matter)
- AussieWog93 5y ago
  >i think it gets rather muddy as there isn't really a good metric for raw signal quality (afaik). there's cell tuning and number of spiking channels, but still not a great measure of snr for bmi work (afaik). often times people will apply measures to the outputs of their systems, like task performance, but part of the problem there is that often the state model has varying quality and suitability to task, so it can be difficult to disambiguate signal quality from state model performance. This was something I picked up on back in 2017. I did manage to come up with a definition of SNR that made some sense (basically the euclidean distance between symbol menas, divided by the noise level along the vector connecting the two symbols, assuming the feature space was basically an N-dimensional QAM signal using features 1...N instead of amplitude and phase) - but even then that didn't take into account the fact that the noise was neither well-approximated by AWGN biased nor even constant... And of course, as you said, you could get a bad SNR just because you're extracting the wrong features (although, to be fair, the same problem can exist in telecoms too).
  - a-dub 5y ago
    hah, it's funny. i come at all of this from a sensorimotor control view (even though i was very interested in speech and language- as for a cs person discrete stuff is easier to reason about) and while i can appreciate ideas like this and the shenoy lab stuff, when i worked on this stuff in practice we had no discrete symbols as we decoded continuous variables like positions, velocities, angles and torques- which didn't seem to have clean mappings into comms/noisy channel theory/info theory.
swordsmith 5y ago
From your top level comment: > a nurse a laminated piece of A4 paper and a patient who can blink get about 15 characters How is this a relevant comparison? 15 characters per minute is much less than the 15 WORDS per minute performance this work demonstrates > I spent a solid 6 months intimately learning the pros and cons of various existing BCI systems, as well as the exact methods researchers use to make their technology look like a breakthrough when it isn't. Then you should know this is a huge deal. > Specifically, here, they've created a system that can decode one of 50 symbols Previous motor and speech neuroprosthetic systems focused on pointing (controlling cursor) and more recently, handwriting (https://www.nature.com/articles/s41586-021-03506-2 https://www.nature.com/articles/s41586-021-03506-2). This work goes gives communication rate similar to the handwriting work, but decoding speech from the motor cortex has been much less understood than that of simple motor movements such as cursor position and velocity control. Even more impressive, the test subject is not even a native English speaker. > They've also cherry-picked the patient that responded the best. Bravo-1 is the only subject they had. > When you peel back the hype, this is about the same performance we've been getting since '08 for an invasive, subdural, non-penetrative array. This is completely, utterly false. See (https://stacks.stanford.edu/file/druid:jx921pv3255/TechnicalReport01_GPT_BCIs-2021-05-20.pdf https://stacks.stanford.edu/file/druid:jx921pv3255/Technical...) for survey of performance of typing BCI. You seem to love to cite your background as a BCI PhD dropout. I should also point out that I have a completed PhD in invasive neuroprosthetics and still work in the field (not that matters when anyone can look up the sources and judge for themselves).
- AussieWog93 5y ago
  >How is this a relevant comparison? 15 characters per minute is much less than the 15 WORDS per minute performance this work demonstrates 15 words from a set of 50 (6 bits per "word" vs. 5 per letter in the English alphabet). It's like saying a 100 baud telegraph machine can decode 300 words per second, just so long as those words come from the set of "dot" and "dash". >but decoding speech from the motor cortex has been much less understood than that of simple motor movements such as cursor position and velocity control. Cursor position and velocity are outputs, the input is still a self-paced motor imagery task. >Bravo-1 is the only subject they had. Fair cop. Maybe they'll get genuinely impressive results with their next patient. >This is completely, utterly false. See (https://stacks.stanford.edu/file/druid:jx921pv3255/Technical https://stacks.stanford.edu/file/druid:jx921pv3255/Technical...) for survey of performance of typing BCI. I posted a link in a comment below showing an ITR of 35bpm from scalp EEG from pre-2010. >I have a completed PhD in invasive neuroprosthetics and still work in the field I am truly sorry for your loss.