8 ms·

Show HN: RoboPianist, a piano playing robot simulation in the browser

260 points by kzakka 4y ago

thallium205 4y ago
Pianist hands and fingering don’t really look like that while playing.
- londons_explore 4y ago
  I think it's because the 38 muscles they have given their robot hand don't map perfectly with real muscles in a real hand. In particular, there seems to be not much ability to move each finger left or right. For example, actuator 14 lets the baby finger swing outwards - but now try to stretch your baby finger out, and you see it can swing much further than this robot finger can swing out, and the pivot is closer to your wrist, allowing more reach.
- sfblah 4y ago
  I can play the songs on the list, and the fingering is completely wrong.
  - kzakka 4y ago
    The fingering we use is taken directly from: https://beam.kisarazu.ac.jp/~saito/research/PianoFingeringDataset/ https://beam.kisarazu.ac.jp/~saito/research/PianoFingeringDa... From their whitepaper: "Fingerings in the dataset were provided by experienced pianists who graduated from a music college or who had played the piano for more than twenty years. The pianists were asked to choose pieces that they could play and provided the fingering that they had actually used for the performance."
    - sfblah 4y ago
      Very interesting. The fingering for the Chopin piece is just bizarre to me, but I'm not going to argue with data.
  - ubj 4y ago
    Fingering isn't a fixed "right" or "wrong" matter. Every pianist's hands are different, so fingerings may vary from person to person.
- londons_explore 4y ago
  I suspect these simulated keys have no weight/spring back force. Ie. the robot hand doesn't have to 'push hard' to make them go down. It would be equivalent to you trying to play a holographic piano in the air in front of you. I suspect you'd adopt a very different hand position too.
  - kzakka 4y ago
    We're using the MuJoCo simulator under the hood: https://mujoco.org/ https://mujoco.org/ The keys are actually implemented using a spring mechanism, but springs in MuJoCo are currently linear, which isn't the case in the real world.
Waterluvian 4y ago
I'm so excited and curious about this that I can't even structure my thoughts well. They're just flooding my brain. What part of this is pre-coded? What part is being generated? Is the goal to give a program some sheet music (maybe a MIDI file) and it figures out the fingerings[1] and then translates the fingerings into kinematics? Because if that's the goal... Holy forking shirtballs that would be amazing. One of the trickiest things for me as a novice pianist is figuring out the fingerings to a piece. It's like a puzzle you work at until you've figured out what's comfortable. It's all about lookahead. "This section generally goes down so I probably want to begin with my pinky and not my thumb." And if it got really good at that, not only are the fingerings useful, but maybe we could get feedback on how physically demanding a piece is. Another challenge I've discovered as a novice is that it can be surprisingly tricky to look at sheet music or hear a piece and determine if it's as easy as it sounds. Some pieces require some very complex fingering. [1] what pianists call the determination of what fingers go where, not just to play certain notes together, but to ensure you can fluidly and comfortably play the next notes as well.
- deleted 4y ago
  [deleted]
- rideontime 4y ago
  clicking the link at the top of the page may help explain :) https://github.com/google-research/robopianist/ https://github.com/google-research/robopianist/
  - ksherlock 4y ago
    That URL also explains why it only works on chrome.
    - kzakka 4y ago
      Hi author here! The app should work on mobile/desktop and was tested on both Safari and Chrome. I've heard it's buggy for some people (unclear if it's an older hardware problem) but you can try this embedded demo which works better: https://kzakka.com/robopianist/#demo https://kzakka.com/robopianist/#demo
deleted 4y ago
[deleted]
ShadowBanThis01 4y ago
Pretty interesting. Just a heads-up: On my system, in desktop Safari, the simulation stalled almost immediately. Upon reload, the fingers moved but no sound came out. After I turned up the volume, I opened the control panel and suddenly the sound started working and blasted me out. Beware! After that it seemed to work OK.
telesilla 4y ago
Oh dear, it's like listening to a 12 year old practicing before their lesson. But hey! Well done Robot for the achievement. I bet in no time you'll be https://youtu.be/e37NxUtFQSo https://youtu.be/e37NxUtFQSo. Good luck with the training!
patrakov 4y ago
Bug: if the WebGL is too slow (e.g. Haswell IGPU at 4K resolution), the song stalls completely. It should still play with reduced fps for rendering. Resizing the window down lets it proceed. Maximizing it again stalls again.
huhtenberg 4y ago
Whoopses shortly after a load with Uncaught Error: buffer is either not set or not loaded ti https://unpkg.com/tone@14.7.77:1 start https://unpkg.com/tone@14.7.77:21 triggerAttack https://unpkg.com/tone@14.7.77:21 triggerAttack https://unpkg.com/tone@14.7.77:21 processPianoState https://kevinzakka.github.io/robopianist-demo/examples/main.js:174 render https://kevinzakka.github.io/robopianist-demo/examples/main.js:255 onAnimationFrame https://kevinzakka.github.io/robopianist-demo/node_modules/three/build/three.module.js:27951 onAnimationFrame https://kevinzakka.github.io/robopianist-demo/node_modules/three/build/three.module.js:12661 This is in a recent Firefox.
- rzzzt 4y ago
  I needed to reload the page once and click on it to start. It might have to do with autoplay blocking.
  - kzakka 4y ago
    Indeed, browsers are pretty aggressive about not autoplaying sound, you need to interact with the screen (mouse click or finger tap + make sure your sound is up / ringer not on silent).
green_man_lives 4y ago
I don't really have a background in ML or anything but how generalizable is this? Is the goal to have a trained model for each specific using this framework? It's pretty amazing, I love seeing anything robotics related in the browser.
reffaelwallen 4y ago
Please add sustain pedal as well, you will get 10x positive reactions
- kzakka 4y ago
  We have the sustain pedal implemented in the standalone MuJoCo simulation, e.g. see https://www.youtube.com/watch?v=VBFn_Gg0yD8 https://www.youtube.com/watch?v=VBFn_Gg0yD8. I just couldn't figure out how to do it with Tone.js :(
- Traubenfuchs 4y ago
  I feel like the sustain pedal is to the piano what the beauty filter is to facebook.
samstave 4y ago
Computer I say, Computer? Uh. you just have to use the _keyboard_ ah, a keyboard, how quaint! - How many words a minute can it type? and while the refinements in movement finess and control are obviously a needed thing, out of ignorance, what other abilities will this allow? I assume that it will allow for a much more finessed touch control of hands/digits, and, coupled with sensors, as they evolve, be able for much more fine crafts - such as embroidery?
abrichr 4y ago
Excellent, thank you for making this open source! I had been playing with the idea of creating a browser-based virtual piano for when I'm travelling and don't have access to a real piano but have my laptop with me. The idea would be to point the webcam down at the table between me and the laptop, and play on the table as if a piano were there. Then use the mediapipe framework [1] to capture finger positions, and use those to update a virtual environment like the one you have here. I put it on hold due to the significant engineering required, but it seems you have already implemented (and open sourced!) the browser-based piano simulation component. A quick scan through your repo indicates that this is all implemented in Python. I see that you are using mujoco_wasm [2]. Can you please comment on what is required to compile your project to work in the browser? Thank you again! [1] https://google.github.io/mediapipe/ https://google.github.io/mediapipe/ [2] https://github.com/zalo/mujoco_wasm https://github.com/zalo/mujoco_wasm
- imposter 4y ago
  That sounds like a great idea, I think you should go ahead and build it because this project is kinda completely different. And mediapipe is surely the way to go for a project idea of yours, since hand-tracking is quite robust with it.
recursive 4y ago
It seems to play with the tops of the fingers rather than the pads of the fingertips at times. In 30 years of playing, I never even thought of trying that. I'm kind of amazed.
- Geee 4y ago
  I use the tops of the fingers when doing glissandos / sliding across the keys. Otherwise, I can't think of a situation where it would make sense.
imposter 4y ago
This is really freakin cool ! As a mechatronics engineer, this is the sort of thing I imagined I would be doing. This really makes me want to dive deep into RL
MarcinQ 4y ago
Hello. I really like this project. It would be awesome if you added option to control fingers with a keyboard.
v9v 4y ago
A similar service: https://www.concertcreator.ai/ https://www.concertcreator.ai/ They claim to be able to generate fingerings and animations from raw audio. The video demos give an idea of the output, but they seem to be shutting down.
moritonal 4y ago
This is cool, but thank you for introducing me to mujoco. This could be exactly the library I've been looking for when it comes to simulating a fencing engine.
jbd0 4y ago
There's no way to adjust tempo?

260 points by kzakka 4y ago

thallium205 4y ago

Pianist hands and fingering don’t really look like that while playing.

londons_explore 4y ago
I think it's because the 38 muscles they have given their robot hand don't map perfectly with real muscles in a real hand. In particular, there seems to be not much ability to move each finger left or right. For example, actuator 14 lets the baby finger swing outwards - but now try to stretch your baby finger out, and you see it can swing much further than this robot finger can swing out, and the pivot is closer to your wrist, allowing more reach.
sfblah 4y ago
I can play the songs on the list, and the fingering is completely wrong.
- kzakka 4y ago
  The fingering we use is taken directly from: https://beam.kisarazu.ac.jp/~saito/research/PianoFingeringDataset/ https://beam.kisarazu.ac.jp/~saito/research/PianoFingeringDa... From their whitepaper: "Fingerings in the dataset were provided by experienced pianists who graduated from a music college or who had played the piano for more than twenty years. The pianists were asked to choose pieces that they could play and provided the fingering that they had actually used for the performance."
  - sfblah 4y ago
    Very interesting. The fingering for the Chopin piece is just bizarre to me, but I'm not going to argue with data.
- ubj 4y ago
  Fingering isn't a fixed "right" or "wrong" matter. Every pianist's hands are different, so fingerings may vary from person to person.
londons_explore 4y ago
I suspect these simulated keys have no weight/spring back force. Ie. the robot hand doesn't have to 'push hard' to make them go down. It would be equivalent to you trying to play a holographic piano in the air in front of you. I suspect you'd adopt a very different hand position too.
- kzakka 4y ago
  We're using the MuJoCo simulator under the hood: https://mujoco.org/ https://mujoco.org/ The keys are actually implemented using a spring mechanism, but springs in MuJoCo are currently linear, which isn't the case in the real world.

Waterluvian 4y ago

I'm so excited and curious about this that I can't even structure my thoughts well. They're just flooding my brain. What part of this is pre-coded? What part is being generated? Is the goal to give a program some sheet music (maybe a MIDI file) and it figures out the fingerings[1] and then translates the fingerings into kinematics? Because if that's the goal... Holy forking shirtballs that would be amazing. One of the trickiest things for me as a novice pianist is figuring out the fingerings to a piece. It's like a puzzle you work at until you've figured out what's comfortable. It's all about lookahead. "This section generally goes down so I probably want to begin with my pinky and not my thumb." And if it got really good at that, not only are the fingerings useful, but maybe we could get feedback on how physically demanding a piece is. Another challenge I've discovered as a novice is that it can be surprisingly tricky to look at sheet music or hear a piece and determine if it's as easy as it sounds. Some pieces require some very complex fingering. [1] what pianists call the determination of what fingers go where, not just to play certain notes together, but to ensure you can fluidly and comfortably play the next notes as well.

deleted 4y ago
[deleted]
rideontime 4y ago
clicking the link at the top of the page may help explain :) https://github.com/google-research/robopianist/ https://github.com/google-research/robopianist/
- ksherlock 4y ago
  That URL also explains why it only works on chrome.
  - kzakka 4y ago
    Hi author here! The app should work on mobile/desktop and was tested on both Safari and Chrome. I've heard it's buggy for some people (unclear if it's an older hardware problem) but you can try this embedded demo which works better: https://kzakka.com/robopianist/#demo https://kzakka.com/robopianist/#demo

deleted 4y ago

[deleted]

ShadowBanThis01 4y ago

Pretty interesting. Just a heads-up: On my system, in desktop Safari, the simulation stalled almost immediately. Upon reload, the fingers moved but no sound came out. After I turned up the volume, I opened the control panel and suddenly the sound started working and blasted me out. Beware! After that it seemed to work OK.

telesilla 4y ago

Oh dear, it's like listening to a 12 year old practicing before their lesson. But hey! Well done Robot for the achievement. I bet in no time you'll be https://youtu.be/e37NxUtFQSo https://youtu.be/e37NxUtFQSo. Good luck with the training!

patrakov 4y ago

Bug: if the WebGL is too slow (e.g. Haswell IGPU at 4K resolution), the song stalls completely. It should still play with reduced fps for rendering. Resizing the window down lets it proceed. Maximizing it again stalls again.

huhtenberg 4y ago

Whoopses shortly after a load with Uncaught Error: buffer is either not set or not loaded ti https://unpkg.com/tone@14.7.77:1 start https://unpkg.com/tone@14.7.77:21 triggerAttack https://unpkg.com/tone@14.7.77:21 triggerAttack https://unpkg.com/tone@14.7.77:21 processPianoState https://kevinzakka.github.io/robopianist-demo/examples/main.js:174 render https://kevinzakka.github.io/robopianist-demo/examples/main.js:255 onAnimationFrame https://kevinzakka.github.io/robopianist-demo/node_modules/three/build/three.module.js:27951 onAnimationFrame https://kevinzakka.github.io/robopianist-demo/node_modules/three/build/three.module.js:12661 This is in a recent Firefox.

rzzzt 4y ago
I needed to reload the page once and click on it to start. It might have to do with autoplay blocking.
- kzakka 4y ago
  Indeed, browsers are pretty aggressive about not autoplaying sound, you need to interact with the screen (mouse click or finger tap + make sure your sound is up / ringer not on silent).

green_man_lives 4y ago

I don't really have a background in ML or anything but how generalizable is this? Is the goal to have a trained model for each specific using this framework? It's pretty amazing, I love seeing anything robotics related in the browser.

reffaelwallen 4y ago

Please add sustain pedal as well, you will get 10x positive reactions

kzakka 4y ago
We have the sustain pedal implemented in the standalone MuJoCo simulation, e.g. see https://www.youtube.com/watch?v=VBFn_Gg0yD8 https://www.youtube.com/watch?v=VBFn_Gg0yD8. I just couldn't figure out how to do it with Tone.js :(
Traubenfuchs 4y ago
I feel like the sustain pedal is to the piano what the beauty filter is to facebook.

samstave 4y ago

Computer I say, Computer? Uh. you just have to use the _keyboard_ ah, a keyboard, how quaint! - How many words a minute can it type? and while the refinements in movement finess and control are obviously a needed thing, out of ignorance, what other abilities will this allow? I assume that it will allow for a much more finessed touch control of hands/digits, and, coupled with sensors, as they evolve, be able for much more fine crafts - such as embroidery?

abrichr 4y ago

Excellent, thank you for making this open source! I had been playing with the idea of creating a browser-based virtual piano for when I'm travelling and don't have access to a real piano but have my laptop with me. The idea would be to point the webcam down at the table between me and the laptop, and play on the table as if a piano were there. Then use the mediapipe framework [1] to capture finger positions, and use those to update a virtual environment like the one you have here. I put it on hold due to the significant engineering required, but it seems you have already implemented (and open sourced!) the browser-based piano simulation component. A quick scan through your repo indicates that this is all implemented in Python. I see that you are using mujoco_wasm [2]. Can you please comment on what is required to compile your project to work in the browser? Thank you again! [1] https://google.github.io/mediapipe/ https://google.github.io/mediapipe/ [2] https://github.com/zalo/mujoco_wasm https://github.com/zalo/mujoco_wasm

imposter 4y ago
That sounds like a great idea, I think you should go ahead and build it because this project is kinda completely different. And mediapipe is surely the way to go for a project idea of yours, since hand-tracking is quite robust with it.

recursive 4y ago

It seems to play with the tops of the fingers rather than the pads of the fingertips at times. In 30 years of playing, I never even thought of trying that. I'm kind of amazed.

Geee 4y ago
I use the tops of the fingers when doing glissandos / sliding across the keys. Otherwise, I can't think of a situation where it would make sense.

imposter 4y ago

This is really freakin cool ! As a mechatronics engineer, this is the sort of thing I imagined I would be doing. This really makes me want to dive deep into RL

MarcinQ 4y ago

Hello. I really like this project. It would be awesome if you added option to control fingers with a keyboard.

v9v 4y ago

A similar service: https://www.concertcreator.ai/ https://www.concertcreator.ai/ They claim to be able to generate fingerings and animations from raw audio. The video demos give an idea of the output, but they seem to be shutting down.

moritonal 4y ago

This is cool, but thank you for introducing me to mujoco. This could be exactly the library I've been looking for when it comes to simulating a fencing engine.

jbd0 4y ago

There's no way to adjust tempo?