7 ms·

Musician Who Lost His Arm Plays Piano Again with AI Prosthesis

88 points by bcaulfield 9y ago

hypersoar 9y ago
This reminds me of Paul Wittgenstein (brother of philospher Ludwig)[1]. He was a concert pianist who lost his right arm in WWI and the commissioned a bunch of piano music playable with only the left hand, including works by still-famous names like Ravel and Prokofiev. My personal favorite is Korngold's [2]. More recently, there is Leon Flescher, who was a highly-renowned pianist until he lost the use of his right hand in the 60s. Amazingly, he got it back 40 years later and has enjoyed recent success again as a two-handed pianist. [1] https://en.wikipedia.org/wiki/Paul_Wittgenstein https://en.wikipedia.org/wiki/Paul_Wittgenstein [2] https://www.youtube.com/watch?v=bGulyl8bzgQ https://www.youtube.com/watch?v=bGulyl8bzgQ
- mc32 9y ago
  Reminds me a bit of the drummer for Def Leopard, learned to drum one handed after car crash left him.with amputated arm.
- Mediterraneo10 9y ago
  A similar case is the Finland-resident Japanese pianist Izumi Tateno, who lost the use of the right half of his body after a stroke and then began giving concerts solely with his left hand.
- JamieBeckett 9y ago
  Wow, that's really interesting. Thanks for sharing this info.
- joshvm 9y ago
  Some famous (ish?) modified versions of Chopin's Etudes were written by Godowsky. His mission appears to have been to make Chopin's hardest pieces just a bit harder, including the Revolutionary for the left hand only: https://youtu.be/X2nMUwdh1Wk?t=2m41s https://youtu.be/X2nMUwdh1Wk?t=2m41s
  - asimpletune 9y ago
    Woah
- jeffwass 9y ago
  I had the pleasure of seeing Nicolas McCarthy speak and play live a couple years ago. He was born with only a right hand, and fought the odds to start learning piano at age 14. He eventually made it into and graduated from the Royal Academy of Music in London, their first one-handed pianist. He plays a variety of left-hand-only repertoire. His playing is amazing, and his story inspirational. http://nicholasmccarthy.co.uk http://nicholasmccarthy.co.uk
dpark 9y ago
Typically these “futuristic” prosthetics are still relying on myoelectric sensing and a lot of wishful thinking. It’s nice to see some meaningful progress in more detailed sensing technologies. This actually seems promising.
- melling 9y ago
  This was a hot thing a few years ago. These guys still seem to be around: https://techcrunch.com/2013/06/05/thalmic-labs-raises-14-5m-to-make-the-myo-armband-the-next-big-thing-in-gesture-control/ https://techcrunch.com/2013/06/05/thalmic-labs-raises-14-5m-... https://www.myo.com https://www.myo.com
- switchbak 9y ago
  In the realm of spinal cord (and nerve repair) this work also struck me as extremely promising: https://www.twu.ca/news-events/news/twu-graduate-creates-nanomaterial-could-help-repair-spinal-cord-injuries https://www.twu.ca/news-events/news/twu-graduate-creates-nan...
bcaulfield 9y ago
Once these are working well will factory workers be able to attach a third and fourth arm for extra productivity?
- osteele 9y ago
  The brain has a couple of regions (the somatosensory context, the primary and supplementary motor cortex) with maps of the body. You'd need to add the limbs to those maps. Brains have enough plasticity to re-allocate the area responsible for limbs or fingers by a minor amount, but nowhere near enough to add a new limb, at least in adulthood.
  - ars 9y ago
    I'm not so sure that's true. People who use a tool for a long time report the tool feeling like an extension of their body. Also, try this thought experiment - try to flap your wings. I know you don't have any, just try to activate them anyway. Do you sense yourself trying to move "something"? Your brain clearly is able to map anything it wants.
    - icegreentea2 9y ago
      For the technology specifically covered in this article, all they are doing is reading the muscle activation of existing muscles, and then mapping to the prosthetic's actuators. The "twist" in this story is that sensing is done with ultrasound instead of EMG (better signal to noise if you can throw enough compute at it), and I believe some cool deep learning tricks to map sets of muscle actuations to prosthetic motion "better". In order for this approach to work, a nervous signal has to travel out of the brain, and at least into the brain stem. In these cases, they actually have to get all the way to a giant amplifier (your muscle) in order to detect it. When your talking about your brain imagining to flap your wings, unless your brain has mapped your swings to your lats, traps and delts, you're not sending any actual motor signals out of your brain and we can't read it. If it is mapped to your lats, traps and delts, then you're actually moving those muscles right now. If we used a system where we tried using your lats, traps and delts as points to control prothestic wings, then you run into the problem of what do you with your wings when you're not trying to use those muscles as a wing controller. In other words, you'll get a modal system... which are tricky. You'd basically have to switch off parts of 'baseline human' functionality at a time to get your 'human+' stuff.
partycoder 9y ago
It is progress, and it is very impressive but the title is a bit misleading. I think there is still a gap between what is being demonstrated to fully "play piano again", in the context of playing piano professionally as a musician. For composition it can be still very valuable.
rjblackman 9y ago
how hard would it be to translate the movements to midi? I'm guessing there would be some way to capture the signals the prosthesis and characterise the mapping from movement to midi.
adrya407 9y ago
What really frustrates me when I watch this kind of demonstrations with bionics is the delay and inaccuracy between desired action and the real effect. I know it's not that simple, but from my perspective since the desired actions are transmitted through our nerves at 100 m/s and we can send an message thousands of kms away in milliseconds, I think there is always space for improvement. I am curious which is the real bottleneck in all the process.
- JamieBeckett 9y ago
  As you can see in the video https://www.youtube.com/watch?v=HjW1kIt5iQg https://www.youtube.com/watch?v=HjW1kIt5iQg there is some latency. Weinberg and Barnes are working to fix that, but I don't know details. Weinberg said he wants to gather more data from more users so the arm is more useful to more people.
  - adrya407 9y ago
    But why should one rely on the termination of the nerves instead of the implating an electrode on the main cortex which controls the motor activity? Can't they translate directly the actions from the energy flow at that level?

88 points by bcaulfield 9y ago

This reminds me of Paul Wittgenstein (brother of philospher Ludwig)[1]. He was a concert pianist who lost his right arm in WWI and the commissioned a bunch of piano music playable with only the left hand, including works by still-famous names like Ravel and Prokofiev. My personal favorite is Korngold's [2]. More recently, there is Leon Flescher, who was a highly-renowned pianist until he lost the use of his right hand in the 60s. Amazingly, he got it back 40 years later and has enjoyed recent success again as a two-handed pianist. [1] https://en.wikipedia.org/wiki/Paul_Wittgenstein https://en.wikipedia.org/wiki/Paul_Wittgenstein [2] https://www.youtube.com/watch?v=bGulyl8bzgQ https://www.youtube.com/watch?v=bGulyl8bzgQ

mc32 9y ago
Reminds me a bit of the drummer for Def Leopard, learned to drum one handed after car crash left him.with amputated arm.
Mediterraneo10 9y ago
A similar case is the Finland-resident Japanese pianist Izumi Tateno, who lost the use of the right half of his body after a stroke and then began giving concerts solely with his left hand.
JamieBeckett 9y ago
Wow, that's really interesting. Thanks for sharing this info.
joshvm 9y ago
Some famous (ish?) modified versions of Chopin's Etudes were written by Godowsky. His mission appears to have been to make Chopin's hardest pieces just a bit harder, including the Revolutionary for the left hand only: https://youtu.be/X2nMUwdh1Wk?t=2m41s https://youtu.be/X2nMUwdh1Wk?t=2m41s
- asimpletune 9y ago
  Woah
jeffwass 9y ago
I had the pleasure of seeing Nicolas McCarthy speak and play live a couple years ago. He was born with only a right hand, and fought the odds to start learning piano at age 14. He eventually made it into and graduated from the Royal Academy of Music in London, their first one-handed pianist. He plays a variety of left-hand-only repertoire. His playing is amazing, and his story inspirational. http://nicholasmccarthy.co.uk http://nicholasmccarthy.co.uk

dpark 9y ago

Typically these “futuristic” prosthetics are still relying on myoelectric sensing and a lot of wishful thinking. It’s nice to see some meaningful progress in more detailed sensing technologies. This actually seems promising.

melling 9y ago
This was a hot thing a few years ago. These guys still seem to be around: https://techcrunch.com/2013/06/05/thalmic-labs-raises-14-5m-to-make-the-myo-armband-the-next-big-thing-in-gesture-control/ https://techcrunch.com/2013/06/05/thalmic-labs-raises-14-5m-... https://www.myo.com https://www.myo.com
switchbak 9y ago
In the realm of spinal cord (and nerve repair) this work also struck me as extremely promising: https://www.twu.ca/news-events/news/twu-graduate-creates-nanomaterial-could-help-repair-spinal-cord-injuries https://www.twu.ca/news-events/news/twu-graduate-creates-nan...

bcaulfield 9y ago

Once these are working well will factory workers be able to attach a third and fourth arm for extra productivity?

osteele 9y ago
The brain has a couple of regions (the somatosensory context, the primary and supplementary motor cortex) with maps of the body. You'd need to add the limbs to those maps. Brains have enough plasticity to re-allocate the area responsible for limbs or fingers by a minor amount, but nowhere near enough to add a new limb, at least in adulthood.
- ars 9y ago
  I'm not so sure that's true. People who use a tool for a long time report the tool feeling like an extension of their body. Also, try this thought experiment - try to flap your wings. I know you don't have any, just try to activate them anyway. Do you sense yourself trying to move "something"? Your brain clearly is able to map anything it wants.
  - icegreentea2 9y ago
    For the technology specifically covered in this article, all they are doing is reading the muscle activation of existing muscles, and then mapping to the prosthetic's actuators. The "twist" in this story is that sensing is done with ultrasound instead of EMG (better signal to noise if you can throw enough compute at it), and I believe some cool deep learning tricks to map sets of muscle actuations to prosthetic motion "better". In order for this approach to work, a nervous signal has to travel out of the brain, and at least into the brain stem. In these cases, they actually have to get all the way to a giant amplifier (your muscle) in order to detect it. When your talking about your brain imagining to flap your wings, unless your brain has mapped your swings to your lats, traps and delts, you're not sending any actual motor signals out of your brain and we can't read it. If it is mapped to your lats, traps and delts, then you're actually moving those muscles right now. If we used a system where we tried using your lats, traps and delts as points to control prothestic wings, then you run into the problem of what do you with your wings when you're not trying to use those muscles as a wing controller. In other words, you'll get a modal system... which are tricky. You'd basically have to switch off parts of 'baseline human' functionality at a time to get your 'human+' stuff.

partycoder 9y ago

It is progress, and it is very impressive but the title is a bit misleading. I think there is still a gap between what is being demonstrated to fully "play piano again", in the context of playing piano professionally as a musician. For composition it can be still very valuable.

rjblackman 9y ago

how hard would it be to translate the movements to midi? I'm guessing there would be some way to capture the signals the prosthesis and characterise the mapping from movement to midi.

adrya407 9y ago

What really frustrates me when I watch this kind of demonstrations with bionics is the delay and inaccuracy between desired action and the real effect. I know it's not that simple, but from my perspective since the desired actions are transmitted through our nerves at 100 m/s and we can send an message thousands of kms away in milliseconds, I think there is always space for improvement. I am curious which is the real bottleneck in all the process.

JamieBeckett 9y ago
As you can see in the video https://www.youtube.com/watch?v=HjW1kIt5iQg https://www.youtube.com/watch?v=HjW1kIt5iQg there is some latency. Weinberg and Barnes are working to fix that, but I don't know details. Weinberg said he wants to gather more data from more users so the arm is more useful to more people.
- adrya407 9y ago
  But why should one rely on the termination of the nerves instead of the implating an electrode on the main cortex which controls the motor activity? Can't they translate directly the actions from the energy flow at that level?