4 ms·

It's funny, I've had almost the same experience: decided to start playing again during Covid, and wanted to use the midi port to gamify things a bit. I've been

It's funny, I've had almost the same experience: decided to start playing again during Covid, and wanted to use the midi port to gamify things a bit. I've been using Synthesia[1] on an iPad, which also lets you take an arbitrary MIDI file and practice it. On the off-chance that this will inspire the article's author for their development, or simply for the curious, this has been my experience with it: The primary mode of display is "Guitar Hero" style (it's hard to describe just look at this video to get an idea[2]) but it's possible to display the sheet music. It lets you practice and grades you depending on how close you're sticking to the score. It's very nice to be able to practice one hand separately while having the app play the other hand. I do fairly well reading sheet music but the first time deciphering it can be a bit tedious, using the app speeds this up considerably for me. Synthesia is almost great but it has a lot of small shortcomings which add up to a bad user experience, for example: - The conversion from MIDI to a readable score is very poor compared to, say, Musescore. The sharps and flats are generally off (e.g. randomly a double flat D will appear instead of a C, even though the score has a C, Musescore interprets the midi as a C, etc). - Not enough wiggle room when playing, e.g. if a chord is not arpeggiated in the midi file, you won't be able to move forward by arpeggiating. - The app really depends on you following the score exactly and linearly. It should also let the score follow you. For example, if you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing. [1] https://synthesiagame.com/ https://synthesiagame.com/ [2] https://www.youtube.com/watch?v=sleZ-hzrtRY https://www.youtube.com/watch?v=sleZ-hzrtRY

microcolonel 6y ago
Aside from the relatively difficult problem of creating scores from MIDI; when practicing piano as a beginner it'd be interesting to have help in positioning/finger selection. I wonder how hard it would be to compute fingering plans (maybe even with a wingspan parameter).
- jacquesm 6y ago
  We are working very hard on solving this in a very interesting and beautiful way. Stay tuned.
  - cscheid 6y ago
    I’m super curious, if you don’t mind my prodding. I looked into doing this for my own version of a JavaScript midi teacher a while back. Any chance you’re doing it via the neural nets sequence to sequence translation algorithms that are super popular in NLP right now? I think that’s a practical way to do it, except for needing a training corpus. I was going to reverse engineer synthesia’s data to try it out, for what’s worth...
    - jacquesm 6y ago
      We do not think that it will need a neural net in order to be able to do this, but we need to prove that in order to make it stick and we're not quite ready to do that. Keep in mind that this is in every sense of the word a side project, we only work on this when we do not have technical due diligence work for customers and that still continues quite strongly. But on 'off' weeks we work on this and the project progresses quite rapidly when we do.
    - microcolonel 6y ago
      I think it could be done with a constraint solver, with a lot fewer cycles.
      - jacquesm 6y ago
        This is the route we are looking at, with some twists.
jacquesm 6y ago
That's an interesting comment. So, the midi->score conversion is indeed one of the harder parts, I spent a lot of time on that and I'm not yet satisfied with the results. There are quite a few ways in which this can be improved further, there is a long list of 'todo's' where I basically have worked out how to make the improvement but I have not yet gotten around to actually implementing it. Midi is 'lossy' in the sense that it flattens a bunch of knowledge about how a piece is to be rendered into note-on/note-off pairs. You then have to do all kinds of tricks to even begin to approach what the score looked originally. Lots of enharmonically (identically sounding) combinations may have looked completely different on the original score, and when you start adding the various chord combinations it gets even more complicated. Even so, I'm strongly convinced it can be solved and we have made considerable progress on this already. The 'wiggle room' aspect of it has been addressed to some extent. For instance, we build up a map of all the notes that are sounding at a given time, and will ignore notes that are technically 'wrong' but that are still sounding. This allows for much smoother restarts after mistakes. Your final point is a bit tricky, but thank you for the hint, I will definitely add it to the list of things to look at to see if we can find a way to make that work.
- murbard2 6y ago
  Musescore does do a great job and is open source, so there's that. Another option is to have users manually upload midi with a proper score attached to some store or community repo with user reviews. It's less satisfying as a solution, but it means all it takes is one motivated user for everyone to enjoy the score. For my final point, you can model this as a hidden markov process. You're observing notes, but not the user moving his mental "cursor". Classical filtering algorithms should do well here. Ah, one more missing feature, good scores have fingering, but it gets lost in the midi. I'm pretty sure fingering can be infered though and this would be pretty powerful. Amusingly, it can also be seen as a filtering problem: - At any given point, a hand can have a finite number of fingerings, and that number isn't that large. Give each of them a difficulty. - Now, make that difficulty conditional on the previous hand position. - Run Viterbi's algorithm on the score to minimize the difficulty. The difficulty can be obtained from an expert or through some physiological model, but they could also be inferred from an existing body of music sheets.
emmanueloga_ 6y ago
I think what's missing on most "piano-games" is that it doesn't really mirror the way I learn a piece. The way I learn is by bars, repeatedly re-playing bars say, 1 to 4, until I get them, them 3 to 7, until I get them, etc.. But most piano apps want you to start playing from start to end slow, then faster, etc. So: piano apps should allow breaking a piece into bars, allow you to repeat those bars again and again until you exceed certain score. Maybe even keep track of which bars are harder to play, and repeating the difficult bars more often, etc.
- profchaos69 6y ago
  I’ve been using piano marvel which does exactly that, and find it pretty great. The only issues with it are that it’s built on somewhat old tech and can be a bit buggy, but for actually learning pieces it is genuinely useful (I’m around grade 7) and also has method/technique pathways you can go through. I think there’s a free trial for it too. (Not affiliated, just something I personally use)
- jacquesm 6y ago
  This one actually does exactly that, so please give it a try. It also has an auto mode where it measures how well you play and then sends you to practice those bars where you are still weak.
  - emmanueloga_ 6y ago
    sg! will take a look!
d883kd8 6y ago
> you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing. I'm a piano noob but wouldn't you want to practice catching back up with the beat if you ever intend to perform?
- thewebcount 6y ago
  This is a really good question, so I don't know why there are downvotes. There are 2 scenarios: 1) you're trying to learn the notes and 2) you're trying to get it in shape for a performance. It sounds like this software is geared towards scenario 1 where you don't yet know the piece and want to learn it. Generally when learning the piece it's best to fix mistakes as they happen, otherwise your brain sort of subconsciously assumes that was what you intended. Once you have the piece down, then when you make a mistake you want to keep going and pretend it never happened because otherwise the rest of the band will leave you behind. (And for soloists, nobody wants to hear you go back and correct a mistake.)
  - jacquesm 6y ago
    Yes, you really don't want to end up 'practicing your mistakes'.
bmn__ 6y ago
Notes should directly align with the keys for usability. Example: https://www.youtube.com/watch?v=6ZJTdQ-xCBE https://www.youtube.com/watch?v=6ZJTdQ-xCBE I'm not quite happy with Synthesia. There is no visual feedback on completing a note. This has been a staple feature in Doremi Mania and Keyboard Mania for twenty years, so they have zero excuse for not implementing it.

microcolonel 6y ago

Aside from the relatively difficult problem of creating scores from MIDI; when practicing piano as a beginner it'd be interesting to have help in positioning/finger selection. I wonder how hard it would be to compute fingering plans (maybe even with a wingspan parameter).

jacquesm 6y ago
We are working very hard on solving this in a very interesting and beautiful way. Stay tuned.
- cscheid 6y ago
  I’m super curious, if you don’t mind my prodding. I looked into doing this for my own version of a JavaScript midi teacher a while back. Any chance you’re doing it via the neural nets sequence to sequence translation algorithms that are super popular in NLP right now? I think that’s a practical way to do it, except for needing a training corpus. I was going to reverse engineer synthesia’s data to try it out, for what’s worth...
  - jacquesm 6y ago
    We do not think that it will need a neural net in order to be able to do this, but we need to prove that in order to make it stick and we're not quite ready to do that. Keep in mind that this is in every sense of the word a side project, we only work on this when we do not have technical due diligence work for customers and that still continues quite strongly. But on 'off' weeks we work on this and the project progresses quite rapidly when we do.
  - microcolonel 6y ago
    I think it could be done with a constraint solver, with a lot fewer cycles.
    - jacquesm 6y ago
      This is the route we are looking at, with some twists.

jacquesm 6y ago

That's an interesting comment. So, the midi->score conversion is indeed one of the harder parts, I spent a lot of time on that and I'm not yet satisfied with the results. There are quite a few ways in which this can be improved further, there is a long list of 'todo's' where I basically have worked out how to make the improvement but I have not yet gotten around to actually implementing it. Midi is 'lossy' in the sense that it flattens a bunch of knowledge about how a piece is to be rendered into note-on/note-off pairs. You then have to do all kinds of tricks to even begin to approach what the score looked originally. Lots of enharmonically (identically sounding) combinations may have looked completely different on the original score, and when you start adding the various chord combinations it gets even more complicated. Even so, I'm strongly convinced it can be solved and we have made considerable progress on this already. The 'wiggle room' aspect of it has been addressed to some extent. For instance, we build up a map of all the notes that are sounding at a given time, and will ignore notes that are technically 'wrong' but that are still sounding. This allows for much smoother restarts after mistakes. Your final point is a bit tricky, but thank you for the hint, I will definitely add it to the list of things to look at to see if we can find a way to make that work.

murbard2 6y ago
Musescore does do a great job and is open source, so there's that. Another option is to have users manually upload midi with a proper score attached to some store or community repo with user reviews. It's less satisfying as a solution, but it means all it takes is one motivated user for everyone to enjoy the score. For my final point, you can model this as a hidden markov process. You're observing notes, but not the user moving his mental "cursor". Classical filtering algorithms should do well here. Ah, one more missing feature, good scores have fingering, but it gets lost in the midi. I'm pretty sure fingering can be infered though and this would be pretty powerful. Amusingly, it can also be seen as a filtering problem: - At any given point, a hand can have a finite number of fingerings, and that number isn't that large. Give each of them a difficulty. - Now, make that difficulty conditional on the previous hand position. - Run Viterbi's algorithm on the score to minimize the difficulty. The difficulty can be obtained from an expert or through some physiological model, but they could also be inferred from an existing body of music sheets.

emmanueloga_ 6y ago

I think what's missing on most "piano-games" is that it doesn't really mirror the way I learn a piece. The way I learn is by bars, repeatedly re-playing bars say, 1 to 4, until I get them, them 3 to 7, until I get them, etc.. But most piano apps want you to start playing from start to end slow, then faster, etc. So: piano apps should allow breaking a piece into bars, allow you to repeat those bars again and again until you exceed certain score. Maybe even keep track of which bars are harder to play, and repeating the difficult bars more often, etc.

profchaos69 6y ago
I’ve been using piano marvel which does exactly that, and find it pretty great. The only issues with it are that it’s built on somewhat old tech and can be a bit buggy, but for actually learning pieces it is genuinely useful (I’m around grade 7) and also has method/technique pathways you can go through. I think there’s a free trial for it too. (Not affiliated, just something I personally use)
jacquesm 6y ago
This one actually does exactly that, so please give it a try. It also has an auto mode where it measures how well you play and then sends you to practice those bars where you are still weak.
- emmanueloga_ 6y ago
  sg! will take a look!

d883kd8 6y ago

> you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing. I'm a piano noob but wouldn't you want to practice catching back up with the beat if you ever intend to perform?

thewebcount 6y ago
This is a really good question, so I don't know why there are downvotes. There are 2 scenarios: 1) you're trying to learn the notes and 2) you're trying to get it in shape for a performance. It sounds like this software is geared towards scenario 1 where you don't yet know the piece and want to learn it. Generally when learning the piece it's best to fix mistakes as they happen, otherwise your brain sort of subconsciously assumes that was what you intended. Once you have the piece down, then when you make a mistake you want to keep going and pretend it never happened because otherwise the rest of the band will leave you behind. (And for soloists, nobody wants to hear you go back and correct a mistake.)
- jacquesm 6y ago
  Yes, you really don't want to end up 'practicing your mistakes'.

bmn__ 6y ago

Notes should directly align with the keys for usability. Example: https://www.youtube.com/watch?v=6ZJTdQ-xCBE https://www.youtube.com/watch?v=6ZJTdQ-xCBE I'm not quite happy with Synthesia. There is no visual feedback on completing a note. This has been a staple feature in Doremi Mania and Keyboard Mania for twenty years, so they have zero excuse for not implementing it.