7 ms·

RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning (2023)

142 points by bemmu 2y ago

fedeb95 2y ago
waiting for Robot Devil.
AIFounder 2y ago
[dead]
kbouck 2y ago
Any way we can get the Westworld intro song into the demo list?
amelius 2y ago
I mean, keys are always in the place where you expect them to be, so what is the challenge here? (Yes, too lazy to read the article).
- rossant 2y ago
  I think the robot learnt to play by itself, right?
- smokel 2y ago
  The idea is that the motor control is pretty hard to program, and by using reinforcement learning, one only has to tell the computer whether it hits the right keys (reward). The reinforcement learning algorithm takes in the state of the fingers and piano keys, and outputs the required motor actions to perform well. Also, as you are too lazy to read an article but still feel entitled to comment on it, I herby give you -1 reward. Let's see if that helps to improve your character.
- cloudbonsai 2y ago
  > what is the challenge here? If you know the precise state of each motor in a robot hand, you can compute the absolute position of its fingertip, right? To make a robot play piano, you need to solve the reverse problem of it i.e. to compute the motor states from the target robot position. This problem is called "Inverse Kinematics" and not fully resolved to this day. Reinforcement Learning (RL) people claim that, if you let a robot move randomly, and give it apporopriate rewards depending on the resulting state, eventually it learns to solve the problem by itself (just like a human baby eventually learns how to walk through trial & error). Now, you are looking at the state of the art of RL in 2023.
  - upbeat_general 2y ago
    The challenge here is not IK. Sure, you don’t need IK if your policy directly operates in joint space but this isn’t the motivation for doing RL. IK to my knowledge is well known in every setting I am aware of.
    - cloudbonsai 2y ago
      > The challenge here is not IK. No, a state-based RL research like this is essentially an IK problem. Given a goal position in the world frame, you need to find out the motor configuration to move your EEF to that goal position. > IK to my knowledge is well known in every setting I am aware of. Really? When I was working in this field, I've actually never seen anyone who used numerical/analytical IK methods on real robots. Granted that I was not a Robotics engineer (I was a Deep Learning engineer in the team), but my impression at the time was that no practical IK solution was available for a robot with 8 DOF, let alone a 20 DOF robot like Shadow Hand.
      - upbeat_general 2y ago
        Here’s the simple reason why this is not the case: If you provide the 6DoF trajectory (+ gripper joints), a lot of robotics (manipulation in particular) is basically solved. The problem is, we don’t have these good trajectories. Sure, joint space is commonly used for learned policies, but cartesian space isn’t uncommon either. IK is really just not a major focus on the learning side of robotics because it’s not the problem. The problem is we don’t know what to do even at the slightly higher level.
- recursive 2y ago
  Where's yours? That's probably the best way to find all the challenges.
davidanekstein 2y ago
Funny enough I’m listening to Rhapsody in Blue while browsing HN. I’d like to see it do that for 17 minutes.
criddell 2y ago
Is this actually buildable? I'd like to hear it on a real piano because (IMHO) the one in the videos sounds bad. I'd like to also hear how loud the mechanical noise of the machine playing the piano would be. Does the left hand work harder with the heavier keys? What would the hands be mounted to?
- j_bum 2y ago
  What do you mean “heavier keys”? Each key (on a good piano) is weighted to have the same force requirements to be played. Of course varying forces can be used to achieve different decibels, but the curve is the same across the piano.
  - ano-ther 2y ago
    Probably this: > Every single key on a grand piano keyboard is weighted differently. This is because the strings for each note are slightly thinner and shorter in the treble register, becoming thicker and longer towards the bass register. As a result, there is greater resistance when playing low notes than when playing high notes. In other words, a heavier touch is required in the left hand and a lighter touch in the right hand. To emulate this in a digital piano, the keys are often individually weighted, with the lower keys heavier than the higher ones — something that’s called graded weighting. https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-weighted-keys/ https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-wei...
    - cbtones 2y ago
      It's not the strings though, the hammers are smaller as you go to the right in the higher register on the piano. The strings are slightly smaller in diameter, but remarkably shorter in length.
- cloudbonsai 2y ago
  > Is this actually buildable Yes. The robot used in this study is Dexterous Hand from Shadow Robot. It's a real product, and costs around $200-300k (for a single hand). Controlling a real robot using a RL policy (model) trained in a simulation environment is also doable. It's called "Sim2Real" and has been widely experimented in the last decade (with mixed success, though).
rossant 2y ago
Nice work! And great interactive 3d application. My 6yo had a lot of fun annoying the robot while it's playing by forcefully moving its hands around.
pama 2y ago
This was a nice demonstration of letting these robotic hands learn to play a keyboard. The technique is limited due to constraints of the hands, and is closer to parts of the technique of early keyboard instruments like an organ or a harpsichord, rather than that of a modern grand piano, which requires a lot more control of the body core, shoulders, elbows, arms, and wrists with the fingers doing as minimal a motion as possible. I suppose a similar algorithm learning to play on a grand piano using a full humanoid body could learn a technique that would be exciting to analyze.
- djtango 2y ago
  Tangential but your comment has just given me a penny drop moment that I can isolate each degree of freedom (finger, wrist, elbow, shoulder) and still play a key to sound a note. For context I have been working on playing multiple lines of music with right hand lately (Chopin etudes, much struggle...) and saw a video from a professional recommending playing the main melody "with the arm". This has helped a lot with the visualisation and now I can really have a cleaner technique playing the accompaniment above the escapement (using nice flat fingers) and the melody with a much richer tone (using the arm typically for the pinky). I sort of understood in an intuitive nonverbal sense what I am doing but after your comment, the mechanics now makes a lot more sense of what I'm actually doing - thanks!
  - pama 2y ago
    Glad it helps. Claudio Arrau thought of the fingers as “dead” with all the piano playing happening using larger muscles. It is a struggle to internalise this concept in some Chopin etudes (op 10, no 2 especially) but it almost immediately helps in others (op 25, no 1).
    - bondarchuk 2y ago
      Interesting, I've read similar things about calligraphy/writing (from the elbow!), hadn't connected it to piano playing.
  - cbtones 2y ago
PatronBernard 2y ago
The thing needs a teacher, terrible technique! /s
- bongodongobob 2y ago
  I think this is worth mentioning for people who don't play piano. It's technique is terrible and bizarre. It's flicking the keys with its fingernails.
- rossant 2y ago
  It plays almost as badly as I do.
beardyw 2y ago
All the right notes but not necessarily at the right time.
- kzrdude 2y ago
  Which is exactly the wrong way around from what's expected for a pianist :)
linwangg 2y ago
[dead]
gtirloni 2y ago
Imediatelly reminded of the anonymous quote: "I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do laundry and dishes." I understand it's a bad argument because 1) people may need assistance to do art/writing and 2) the advancements gained from teaching AI to do art can be applied to other non-artistic endeavors (e.g. piano AI could be really good to operate machines with lots of buttons and no computerized interface). However, the cost cutting side of the argument is the one that bothers me because companies/people WILL use AI like that in place of actual humans because they're likely to be cheaper in the future. So that pianist or musician playing in a local restaurant can be sure their job will be automated away by a subpar AI and real humans will be relegated to very expensive locations (an extension of replacing humans with recorded music, in a way). My pessismist side thinks greed will be the downfall of humanity.
- ebiester 2y ago
  I agree, but live performance in the age of recording is already a luxury good, and it is unlikely an AI would replace live performance any more than a player piano or CD player would (or has.)
  - dingnuts 2y ago
    I can't see how in a world full of recordings how in person, live performance art could get anything but vastly more valuable. I saw a stage performance of Clue a few weeks ago. It was lovely. Nothing was AI generated, and couldn't be, because it was people physically making art live. How nice. I will continue to pay for those experiences and to prioritize them over recorded ones.
    - gtirloni 2y ago
      No doubt it'll be more valuable but the number of opportunities (to watch and play at those) will decrease globally.
- bluGill 2y ago
  The pianist playing in a restaurant was optimized by the player piano (which would only play if you put a nickel in) back in the late 1800s. Muzak put the final nail in that coffin in 1954. The player piano technology comes from the barrel organ, which seems to have been on the first pipe organs.
pacificmaelstrm 2y ago
"Player Piano"
Tycho 2y ago
Has anyone tried to get a robot to do oil paintings?
- adamheins 2y ago
  This company: https://www.acrylicrobotics.ca/ https://www.acrylicrobotics.ca/
deleted 2y ago
[deleted]
torusle 2y ago
Honestly, this is really bad. It might be a breakthrough of what you are doing, but when I listen to the output all of the timing and phrasing is aweful.
- scientaster2 2y ago
  Alright then, post your ML simulation of a piano player, I'll wait :)
- deskr 2y ago
  [flagged]
ericye16 2y ago
This was the basis of a project I did for my deep reinforcement learning class! https://ericye16.com/stanford-cs224r https://ericye16.com/stanford-cs224r We were able to make some improvements by tuning how the reward is distributed and also by first pretraining the agent on scales before fine-tuning them on the final pieces. Thanks to Kevin Zakka for helping us get started with the RL environment!
- plaguuuuuu 2y ago
  did you guys ever try having the agents play the song slower at first?
  - ericye16 2y ago
    We definitely tried extending the lookahead, but I don't think we tried having a curriculum-style thing where we gradually increased the speed of the song. Great idea though!
cytocync 2y ago
[dead]
wbakst 2y ago
the fact that the failure modes come from limitations of the robotic hands is super interesting
kittikitti 2y ago
> UC Berkeley, Google DeepMind, Stanford University I've personally experienced how research around this time was being shut down because of AI doomerism. People were getting laid off because of it. It's clear to me that these institutions actively spread AI doomerism so that they have full control over it. They actively called for a stop in AI research so that their personal labs can leap forward ahead. It was a little too on the nose for Big Tech, but they don't understand nuance.
iwontberude 2y ago
Wasted opportunity to not play the Westworld Theme
marbro 2y ago
How does this research help us win naval battles? Should Elon and his DOGE friends cut it?

142 points by bemmu 2y ago

fedeb95 2y ago

waiting for Robot Devil.

AIFounder 2y ago

[dead]

kbouck 2y ago

Any way we can get the Westworld intro song into the demo list?

amelius 2y ago

I mean, keys are always in the place where you expect them to be, so what is the challenge here? (Yes, too lazy to read the article).

rossant 2y ago
I think the robot learnt to play by itself, right?
smokel 2y ago
The idea is that the motor control is pretty hard to program, and by using reinforcement learning, one only has to tell the computer whether it hits the right keys (reward). The reinforcement learning algorithm takes in the state of the fingers and piano keys, and outputs the required motor actions to perform well. Also, as you are too lazy to read an article but still feel entitled to comment on it, I herby give you -1 reward. Let's see if that helps to improve your character.
cloudbonsai 2y ago
> what is the challenge here? If you know the precise state of each motor in a robot hand, you can compute the absolute position of its fingertip, right? To make a robot play piano, you need to solve the reverse problem of it i.e. to compute the motor states from the target robot position. This problem is called "Inverse Kinematics" and not fully resolved to this day. Reinforcement Learning (RL) people claim that, if you let a robot move randomly, and give it apporopriate rewards depending on the resulting state, eventually it learns to solve the problem by itself (just like a human baby eventually learns how to walk through trial & error). Now, you are looking at the state of the art of RL in 2023.
- upbeat_general 2y ago
  The challenge here is not IK. Sure, you don’t need IK if your policy directly operates in joint space but this isn’t the motivation for doing RL. IK to my knowledge is well known in every setting I am aware of.
  - cloudbonsai 2y ago
    > The challenge here is not IK. No, a state-based RL research like this is essentially an IK problem. Given a goal position in the world frame, you need to find out the motor configuration to move your EEF to that goal position. > IK to my knowledge is well known in every setting I am aware of. Really? When I was working in this field, I've actually never seen anyone who used numerical/analytical IK methods on real robots. Granted that I was not a Robotics engineer (I was a Deep Learning engineer in the team), but my impression at the time was that no practical IK solution was available for a robot with 8 DOF, let alone a 20 DOF robot like Shadow Hand.
    - upbeat_general 2y ago
      Here’s the simple reason why this is not the case: If you provide the 6DoF trajectory (+ gripper joints), a lot of robotics (manipulation in particular) is basically solved. The problem is, we don’t have these good trajectories. Sure, joint space is commonly used for learned policies, but cartesian space isn’t uncommon either. IK is really just not a major focus on the learning side of robotics because it’s not the problem. The problem is we don’t know what to do even at the slightly higher level.
recursive 2y ago
Where's yours? That's probably the best way to find all the challenges.

davidanekstein 2y ago

Funny enough I’m listening to Rhapsody in Blue while browsing HN. I’d like to see it do that for 17 minutes.

criddell 2y ago

Is this actually buildable? I'd like to hear it on a real piano because (IMHO) the one in the videos sounds bad. I'd like to also hear how loud the mechanical noise of the machine playing the piano would be. Does the left hand work harder with the heavier keys? What would the hands be mounted to?

j_bum 2y ago
What do you mean “heavier keys”? Each key (on a good piano) is weighted to have the same force requirements to be played. Of course varying forces can be used to achieve different decibels, but the curve is the same across the piano.
- ano-ther 2y ago
  Probably this: > Every single key on a grand piano keyboard is weighted differently. This is because the strings for each note are slightly thinner and shorter in the treble register, becoming thicker and longer towards the bass register. As a result, there is greater resistance when playing low notes than when playing high notes. In other words, a heavier touch is required in the left hand and a lighter touch in the right hand. To emulate this in a digital piano, the keys are often individually weighted, with the lower keys heavier than the higher ones — something that’s called graded weighting. https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-weighted-keys/ https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-wei...
  - cbtones 2y ago
    It's not the strings though, the hammers are smaller as you go to the right in the higher register on the piano. The strings are slightly smaller in diameter, but remarkably shorter in length.
cloudbonsai 2y ago
> Is this actually buildable Yes. The robot used in this study is Dexterous Hand from Shadow Robot. It's a real product, and costs around $200-300k (for a single hand). Controlling a real robot using a RL policy (model) trained in a simulation environment is also doable. It's called "Sim2Real" and has been widely experimented in the last decade (with mixed success, though).

rossant 2y ago

Nice work! And great interactive 3d application. My 6yo had a lot of fun annoying the robot while it's playing by forcefully moving its hands around.

pama 2y ago

This was a nice demonstration of letting these robotic hands learn to play a keyboard. The technique is limited due to constraints of the hands, and is closer to parts of the technique of early keyboard instruments like an organ or a harpsichord, rather than that of a modern grand piano, which requires a lot more control of the body core, shoulders, elbows, arms, and wrists with the fingers doing as minimal a motion as possible. I suppose a similar algorithm learning to play on a grand piano using a full humanoid body could learn a technique that would be exciting to analyze.

djtango 2y ago
Tangential but your comment has just given me a penny drop moment that I can isolate each degree of freedom (finger, wrist, elbow, shoulder) and still play a key to sound a note. For context I have been working on playing multiple lines of music with right hand lately (Chopin etudes, much struggle...) and saw a video from a professional recommending playing the main melody "with the arm". This has helped a lot with the visualisation and now I can really have a cleaner technique playing the accompaniment above the escapement (using nice flat fingers) and the melody with a much richer tone (using the arm typically for the pinky). I sort of understood in an intuitive nonverbal sense what I am doing but after your comment, the mechanics now makes a lot more sense of what I'm actually doing - thanks!
- pama 2y ago
  Glad it helps. Claudio Arrau thought of the fingers as “dead” with all the piano playing happening using larger muscles. It is a struggle to internalise this concept in some Chopin etudes (op 10, no 2 especially) but it almost immediately helps in others (op 25, no 1).
  - bondarchuk 2y ago
    Interesting, I've read similar things about calligraphy/writing (from the elbow!), hadn't connected it to piano playing.
- cbtones 2y ago

PatronBernard 2y ago

The thing needs a teacher, terrible technique! /s

bongodongobob 2y ago
I think this is worth mentioning for people who don't play piano. It's technique is terrible and bizarre. It's flicking the keys with its fingernails.
rossant 2y ago
It plays almost as badly as I do.

beardyw 2y ago

All the right notes but not necessarily at the right time.

kzrdude 2y ago
Which is exactly the wrong way around from what's expected for a pianist :)

linwangg 2y ago

[dead]

gtirloni 2y ago

Imediatelly reminded of the anonymous quote: "I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do laundry and dishes." I understand it's a bad argument because 1) people may need assistance to do art/writing and 2) the advancements gained from teaching AI to do art can be applied to other non-artistic endeavors (e.g. piano AI could be really good to operate machines with lots of buttons and no computerized interface). However, the cost cutting side of the argument is the one that bothers me because companies/people WILL use AI like that in place of actual humans because they're likely to be cheaper in the future. So that pianist or musician playing in a local restaurant can be sure their job will be automated away by a subpar AI and real humans will be relegated to very expensive locations (an extension of replacing humans with recorded music, in a way). My pessismist side thinks greed will be the downfall of humanity.

ebiester 2y ago
I agree, but live performance in the age of recording is already a luxury good, and it is unlikely an AI would replace live performance any more than a player piano or CD player would (or has.)
- dingnuts 2y ago
  I can't see how in a world full of recordings how in person, live performance art could get anything but vastly more valuable. I saw a stage performance of Clue a few weeks ago. It was lovely. Nothing was AI generated, and couldn't be, because it was people physically making art live. How nice. I will continue to pay for those experiences and to prioritize them over recorded ones.
  - gtirloni 2y ago
    No doubt it'll be more valuable but the number of opportunities (to watch and play at those) will decrease globally.
bluGill 2y ago
The pianist playing in a restaurant was optimized by the player piano (which would only play if you put a nickel in) back in the late 1800s. Muzak put the final nail in that coffin in 1954. The player piano technology comes from the barrel organ, which seems to have been on the first pipe organs.

pacificmaelstrm 2y ago

"Player Piano"

Tycho 2y ago

Has anyone tried to get a robot to do oil paintings?

adamheins 2y ago
This company: https://www.acrylicrobotics.ca/ https://www.acrylicrobotics.ca/

deleted 2y ago

[deleted]

torusle 2y ago

Honestly, this is really bad. It might be a breakthrough of what you are doing, but when I listen to the output all of the timing and phrasing is aweful.

scientaster2 2y ago
Alright then, post your ML simulation of a piano player, I'll wait :)
deskr 2y ago
[flagged]

ericye16 2y ago

This was the basis of a project I did for my deep reinforcement learning class! https://ericye16.com/stanford-cs224r https://ericye16.com/stanford-cs224r We were able to make some improvements by tuning how the reward is distributed and also by first pretraining the agent on scales before fine-tuning them on the final pieces. Thanks to Kevin Zakka for helping us get started with the RL environment!

plaguuuuuu 2y ago
did you guys ever try having the agents play the song slower at first?
- ericye16 2y ago
  We definitely tried extending the lookahead, but I don't think we tried having a curriculum-style thing where we gradually increased the speed of the song. Great idea though!

cytocync 2y ago

[dead]

wbakst 2y ago

the fact that the failure modes come from limitations of the robotic hands is super interesting

kittikitti 2y ago

> UC Berkeley, Google DeepMind, Stanford University I've personally experienced how research around this time was being shut down because of AI doomerism. People were getting laid off because of it. It's clear to me that these institutions actively spread AI doomerism so that they have full control over it. They actively called for a stop in AI research so that their personal labs can leap forward ahead. It was a little too on the nose for Big Tech, but they don't understand nuance.

iwontberude 2y ago

Wasted opportunity to not play the Westworld Theme

marbro 2y ago

How does this research help us win naval battles? Should Elon and his DOGE friends cut it?