17 ms·

Researchers’ AI aligns sheet music with MIDI audio

97 points by Bud 6y ago

neolog 6y ago
If you already have a MIDI why would you need the sheet music?
- klodolph 6y ago
  MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.
  - neolog 6y ago
    What information isn't captured in the MIDI but is captured in the electronic sheet music?
    - rajangdavis 6y ago
      Sheet music is human readable, MIDI is for computers.
    - BookPage 6y ago
      There's a lot of "annotations"[0] in sheet music which allow the performer to use their judgement for playing. Sure you can extract these patterns from MIDI based on things like velocity curves, but by default raw MIDI won't tell you that. [0] https://en.wikipedia.org/wiki/Dynamics_(music) https://en.wikipedia.org/wiki/Dynamics_(music)
    - deleted 6y ago
      [deleted]
    - modeless 6y ago
      It's more the other way around. The MIDI has extra information that obfuscates the original sheet music. Most commonly, timing variation which can be intentional and quite large, but also other kinds of variation and even mistakes. Additionally, the system described in the article works on scanned paper sheet music, not electronic sheet music, so you also have to factor in OCR issues. All things considered it would be very difficult to manually write an algorithm to do this reliably.
      - PaulDavisThe1st 6y ago
        technically, that would be a case of the MIDI data having noise in it, not extra information.
        kevingadd 6y ago
        It seems unreasonable to call something "noise" if it's meant to be kept. Most traditional definitions of that word would not apply to things like syncopation and variation
        PaulDavisThe1st 6y ago
        "Timing variations" as mentioned in the comment I was replying to was, in context, "noise". If the performer was adding syncopation or other "planned" variation, then sure, it's information, but I don't think that is what was meant.
        modeless 6y ago
        That is what was meant, in part, as indicated by the comment "which can be intentional". And even unintentional variation can be desired, as it makes the sound less robotic. Noise can be information too.
        PaulDavisThe1st 6y ago
        My reading of this was different. I felt that the commenter was noting how the timing information in MIDI is not restricted to the note types represented in sheet music, and can include swing and shuffle that could not be represented in that format. If those timing variations were intentional, then I agree they represent information. If they are just sloppy playing, they are noise. And if the performer is just applying a stylistic form ("swing these bars"), then it's an inefficient encoding of the musical intent.
    - klodolph 6y ago
      Tons. Nearly everything. Sheet music is very expressive. So is MIDI. It’s kind of like asking, “What information isn’t captured in the movie but is captured in the novel?” They’re just radically different ways of expressing music, and much is lost when you translate from one to the other. If you are curious, try flipping through a book on orchestration at the library sometime, or a book on music theory. Even at its most basic, a score records which notes to play, but MIDI doesn’t even do that—MIDI records the note values only, so you can’t tell the difference between G# and Ab. Then add in all the articulations, dynamics, and arbitrary instructions that a composer can put into a score.
      - PaulDavisThe1st 6y ago
        Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). Obviously "arbitrary instructions" are out, but then they are also not formally part of "sheet music", but more "additional written material from the composer", which can accompany MIDI in various ways too.
        neolog 6y ago
        > Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?
        PaulDavisThe1st 6y ago
        Not relevant to the line quoted above, but ... you can apply more or less bow pressure, resulting in different timbres. You can pick/pluck the strings rather than bowing them at all.
        bluGill 6y ago
    - tolbish 6y ago
      Sheet music always has room for human interpretation.
    - powersnail 6y ago
      Dynamics. MIDI has volume, but that's not all there is. Legato vs detache. Articulation. All sorts of esoteric annotations, e.g. "Fire the cannon!", or "Release the penguins."
    - marcan_42 6y ago
      MIDI was designed around the piano, which is a very low-dimensional instrument. Due to how it works (hammers that hit the string with a given physical velocity), the only things that affect how a note sounds are: when you play it, what key you play, how hard you hit it, and the exact position of the damper pedal (and other pedals if used)%. Pianos have a mechanical system that abstracts your playing and confines it to these dimensions. These are the exact dimensions that MIDI covers. And this is why these days we have extremely accurate piano samples as well as physical modeling synths that sound amazing with MIDI input. However, on other instruments, you have a much more direct relationship with the sound. With a violin or guitar, you physically touch the strings - directly or with some device. That already means you have something like 6 degrees of freedom in physical space, not counting fretting. Plus if you use a finger directly, your playing won't sound the same as someone else's with different fingers. The whole action of playing a note is a complex process in time, not a single event, so you can't capture it as one set of parameters. Then there's things like tapping on the body of the instrument, not the strings. You'd basically need a model of the contact surface that touches the string, its physical parameters, and its precise motion in time at high resolution, to accurately capture all possible playing techniques that people actually use. What we've done with sheet music is standardize many of these techniques into instructions that players can perform. It's obviously only a much lower information version, but it's there. Of course, if your question is "can we encode everything in sheet music with MIDI" the answer is obviously yes, but it's not standardized. A given virtual instrument could hypothetically implement anything you could write a score for (perhaps with an obscene number of samples or a really clever modeling synths) and be controlled by MIDI, but only a tiny subset of these options have vaguely standardized mappings to MIDI. MIDI is a very limited standard with tons of room for expansion, and only the most basic MIDI features interoperate between manufacturers. You can make MIDI music sound amazing with specific setups (and this is effectively how a lot of modern music is produced), but you can't make General MIDI music (the standardized subset) sound amazing. The other issue is that sheet music is at a higher abstraction level than MIDI. It leaves more to the interpretation of the performer, but it's designed to be easily interpertable. For the portions of this that MIDI does express at a lower level, it's very hard to infer the higher level representation. % This is excluding prepared pianos and techniques like reaching in and touching the strings, which are rare; of course technically you can do crazy things to a piano, but in practice we don't.
- breakfastduck 6y ago
  You may have composed something you need to provide to classic musicians to perform
- yowlingcat 6y ago
  Think of it as the difference between a computer algebra system and LaTeX. Sure, they're rudimentarily both about mathematics. But the former is about building software that can do interesting computations with algebraic expressions. The latter is about building graphical representations expressly for humans. That's the difference between sheet music and MIDI. MIDI is extremely powerful, but it's not really about setting not just instructions but guidelines for human performers who will ultimately use their judgment on how to interpret it and perform it -- often in large groups. Compared to that, MIDI is not just way more low level, but simply for a difference purpose.
- matheist 6y ago
  See this thread for an example: https://news.ycombinator.com/item?id=24193405 https://news.ycombinator.com/item?id=24193405
- Kye 6y ago
  Handing a MIDI printout to a musician who expects sheet music would be like handing a novel to a movie director who expects a script.
- ksaj 6y ago
  The first example I can think of, because I run into it often, is when you use syncopation. It's made even worse if the syncopated line contains chords, and then even more-so if the note lengths within the chord are not robotically static. Writing that on the MIDI piano roll is slobber proof. But if you take music written that way and view it as sheet music, it is nearly impossible to read because of the many ties that don't even necessarily begin or even end in the same place as the others. It all ends up looking like layered sheets of snow sliding off your roof. A ribbon of differently-sagging tie-lines making it not only unsightly, but impossible to read. Why sheet music if you have MIDI already? Because as someone who records, I use MIDI for writing, but I don't expect other musicians to follow my piano rolls when recording guitar, bass, vocals, etc. and I wouldn't want them ever to sound as robotic as MIDI tends to be. Drums however - that's one area where MIDI piano roll to sheet music actually works flawlessly, even if the notes don't have the usual x| appearance.
  - flipcoder 6y ago
    Does your midi software properly translate drum rudiments that have specific notation symbols? That would be quite impressive actually. Drumming has a lot of technique-specific and informal stylistic things that would be hard to translate.
- amelius 6y ago
  Perhaps they used MIDI only for reference during development. In an actual application they wouldn't need the MIDI. Just my guess.
  - pishpash 6y ago
    If anyone bothered to click the link, it's not MIDI but MIDI-generated audio (title should be changed). They used MIDI audio because likely it simplified the problem along a number of dimensions: rhythm and timbre are fixed, and there is no noise. It also means it's cheating and it will not work on acoustic performances, but it's a good start. I mean if you can't solve the problem for MIDI audio you will never solve it for anything else.
adenadel 6y ago
Intuitively (to me), it feels like a clever variation of something like Needleman-Wunsch could pull this off without using machine learning (as long as you have the right encodings). Can someone enlighten me?
- klodolph 6y ago
  Sheet music is a sequence of images. Maybe you thought sheet music was more like a text file?
  - adenadel 6y ago
    I should have been more clear. This is what I meant be "correct encodings". I would be surprised (but am welcome to being wrong) if a large amount of sheet music hadn't been encoded somehow.
    - klodolph 6y ago
      “Digitized” just means PDF, at best.
    - mjburgess 6y ago
      Sheet music isn't mathematical, ie., there isn't a numerical "encoding". It's has a fair amount of natural semantics (eg., "play like a fantasy"), There is only an embedding or re-presentation in some less informative form. ie., you have to choose some partial measurement system and apply this to the images of scores to produce a numeric representation. This is, in part, why a midi-file is always going to sound worse than an orchestra operating from sheet music. The musicians understand the intensions of the composer.
- yorwba 6y ago
  The sheet music is input as an image, which is not the right encoding for comparison with MIDI, so you need some kind of image recognition in the pipeline. I do think that they could replace the recurrent part of their network by dynamic programming and only use machine learning for feature extraction.
swiley 6y ago
Does this really need AI? Couldn't you just take the maximum of the fourier transform of the rendering multiplied with the audio? Maybe across multiple dimensions.
- klodolph 6y ago
  This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.
  - iainctduncan 6y ago
    And that's before even separating out by instrument! It's really quite incredible what goes on between the ear and the brain for this kind of thing.
    - pishpash 6y ago
      Correlative methods are the wrong model by themselves but could be one part of the right model. Cognition happens by correlation, but also by denoising, by adaptive filtering, by interference cancellation, even by predictive generation. Listening is not a passive process.
  - creato 6y ago
    There's a game called Rocksmith that has been on the market for many years that basically does this in real time on live guitar playing with far more than enough accuracy to make the game fun. And that is doing it basically per-note! This task (aligning sheet music with MIDI) would allow for global optimization approaches which should be far easier and more likely to work.
    - kevingadd 6y ago
      Electric guitar going through a clean (ish) input is a pretty narrowly scoped problem that won't necessarily generalize
deleted 6y ago
[deleted]
gnrlst 6y ago
I'm going through various books by Arnold Schoenberg. He adds lots of tiny examples that I dont have the skill to play nor the ear to "listen" to in my head. HN: Is there a tool/service/app that allows me to point at a piece of sheet music and it just plays it? I've tried a couple apps but they don't work well.
natn 6y ago
What I don't get is why the emphasis on sheet music as an image? I get that mapping from an auditory stream onto a score is an interesting problem, but isn't it naturally decoupled from the visual processing problem of reading and segmenting images of sheet music? Wouldn't it make more sense to deal primarily with a digital intermediate encoding of the score? What am I missing?
bane 6y ago
I wonder how this performs on Black MIDI https://www.youtube.com/watch?v=tfrIXiemYqo&ab_channel=Gingeas https://www.youtube.com/watch?v=tfrIXiemYqo&ab_channel=Ginge...

97 points by Bud 6y ago

neolog 6y ago

If you already have a MIDI why would you need the sheet music?

klodolph 6y ago
MIDI is a very poor substitute for sheet music, and vice versa. Think of MIDI as a performance and sheet music as the text being performed. If you want to analyze the music or perform it, you want the sheet music. If you want a computer to perform the music, you want the MIDI.
- neolog 6y ago
  What information isn't captured in the MIDI but is captured in the electronic sheet music?
  - rajangdavis 6y ago
    Sheet music is human readable, MIDI is for computers.
  - BookPage 6y ago
    There's a lot of "annotations"[0] in sheet music which allow the performer to use their judgement for playing. Sure you can extract these patterns from MIDI based on things like velocity curves, but by default raw MIDI won't tell you that. [0] https://en.wikipedia.org/wiki/Dynamics_(music) https://en.wikipedia.org/wiki/Dynamics_(music)
  - deleted 6y ago
    [deleted]
  - modeless 6y ago
    It's more the other way around. The MIDI has extra information that obfuscates the original sheet music. Most commonly, timing variation which can be intentional and quite large, but also other kinds of variation and even mistakes. Additionally, the system described in the article works on scanned paper sheet music, not electronic sheet music, so you also have to factor in OCR issues. All things considered it would be very difficult to manually write an algorithm to do this reliably.
    - PaulDavisThe1st 6y ago
      technically, that would be a case of the MIDI data having noise in it, not extra information.
      - kevingadd 6y ago
        It seems unreasonable to call something "noise" if it's meant to be kept. Most traditional definitions of that word would not apply to things like syncopation and variation
        PaulDavisThe1st 6y ago
        "Timing variations" as mentioned in the comment I was replying to was, in context, "noise". If the performer was adding syncopation or other "planned" variation, then sure, it's information, but I don't think that is what was meant.
        modeless 6y ago
        That is what was meant, in part, as indicated by the comment "which can be intentional". And even unintentional variation can be desired, as it makes the sound less robotic. Noise can be information too.
        PaulDavisThe1st 6y ago
        My reading of this was different. I felt that the commenter was noting how the timing information in MIDI is not restricted to the note types represented in sheet music, and can include swing and shuffle that could not be represented in that format. If those timing variations were intentional, then I agree they represent information. If they are just sloppy playing, they are noise. And if the performer is just applying a stylistic form ("swing these bars"), then it's an inefficient encoding of the musical intent.
  - klodolph 6y ago
    Tons. Nearly everything. Sheet music is very expressive. So is MIDI. It’s kind of like asking, “What information isn’t captured in the movie but is captured in the novel?” They’re just radically different ways of expressing music, and much is lost when you translate from one to the other. If you are curious, try flipping through a book on orchestration at the library sometime, or a book on music theory. Even at its most basic, a score records which notes to play, but MIDI doesn’t even do that—MIDI records the note values only, so you can’t tell the difference between G# and Ab. Then add in all the articulations, dynamics, and arbitrary instructions that a composer can put into a score.
    - PaulDavisThe1st 6y ago
      Depends a lot on the instrument. For a keyboard instrument, played conventionally, MIDI does a fairly good job of capturing everything that a composer could tell a performer (and vice-versa). Slightly less true for percussion, but still somewhat true. Rather untrue for any instrument where technique can be (ab)used to alter timbre significantly (e.g. most reeds, most strings) Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). Obviously "arbitrary instructions" are out, but then they are also not formally part of "sheet music", but more "additional written material from the composer", which can accompany MIDI in various ways too.
      - neolog 6y ago
        > Very untrue for any instrument capable of continuous pitch generation (e.g. unfretted strings). What are the dimensions of a note that can be varied on a violin other than pitch and loudness?
        PaulDavisThe1st 6y ago
        Not relevant to the line quoted above, but ... you can apply more or less bow pressure, resulting in different timbres. You can pick/pluck the strings rather than bowing them at all.
        bluGill 6y ago
  - tolbish 6y ago
    Sheet music always has room for human interpretation.
  - powersnail 6y ago
    Dynamics. MIDI has volume, but that's not all there is. Legato vs detache. Articulation. All sorts of esoteric annotations, e.g. "Fire the cannon!", or "Release the penguins."
  - marcan_42 6y ago
    MIDI was designed around the piano, which is a very low-dimensional instrument. Due to how it works (hammers that hit the string with a given physical velocity), the only things that affect how a note sounds are: when you play it, what key you play, how hard you hit it, and the exact position of the damper pedal (and other pedals if used)%. Pianos have a mechanical system that abstracts your playing and confines it to these dimensions. These are the exact dimensions that MIDI covers. And this is why these days we have extremely accurate piano samples as well as physical modeling synths that sound amazing with MIDI input. However, on other instruments, you have a much more direct relationship with the sound. With a violin or guitar, you physically touch the strings - directly or with some device. That already means you have something like 6 degrees of freedom in physical space, not counting fretting. Plus if you use a finger directly, your playing won't sound the same as someone else's with different fingers. The whole action of playing a note is a complex process in time, not a single event, so you can't capture it as one set of parameters. Then there's things like tapping on the body of the instrument, not the strings. You'd basically need a model of the contact surface that touches the string, its physical parameters, and its precise motion in time at high resolution, to accurately capture all possible playing techniques that people actually use. What we've done with sheet music is standardize many of these techniques into instructions that players can perform. It's obviously only a much lower information version, but it's there. Of course, if your question is "can we encode everything in sheet music with MIDI" the answer is obviously yes, but it's not standardized. A given virtual instrument could hypothetically implement anything you could write a score for (perhaps with an obscene number of samples or a really clever modeling synths) and be controlled by MIDI, but only a tiny subset of these options have vaguely standardized mappings to MIDI. MIDI is a very limited standard with tons of room for expansion, and only the most basic MIDI features interoperate between manufacturers. You can make MIDI music sound amazing with specific setups (and this is effectively how a lot of modern music is produced), but you can't make General MIDI music (the standardized subset) sound amazing. The other issue is that sheet music is at a higher abstraction level than MIDI. It leaves more to the interpretation of the performer, but it's designed to be easily interpertable. For the portions of this that MIDI does express at a lower level, it's very hard to infer the higher level representation. % This is excluding prepared pianos and techniques like reaching in and touching the strings, which are rare; of course technically you can do crazy things to a piano, but in practice we don't.
breakfastduck 6y ago
You may have composed something you need to provide to classic musicians to perform
yowlingcat 6y ago
Think of it as the difference between a computer algebra system and LaTeX. Sure, they're rudimentarily both about mathematics. But the former is about building software that can do interesting computations with algebraic expressions. The latter is about building graphical representations expressly for humans. That's the difference between sheet music and MIDI. MIDI is extremely powerful, but it's not really about setting not just instructions but guidelines for human performers who will ultimately use their judgment on how to interpret it and perform it -- often in large groups. Compared to that, MIDI is not just way more low level, but simply for a difference purpose.
matheist 6y ago
See this thread for an example: https://news.ycombinator.com/item?id=24193405 https://news.ycombinator.com/item?id=24193405
Kye 6y ago
Handing a MIDI printout to a musician who expects sheet music would be like handing a novel to a movie director who expects a script.
ksaj 6y ago
The first example I can think of, because I run into it often, is when you use syncopation. It's made even worse if the syncopated line contains chords, and then even more-so if the note lengths within the chord are not robotically static. Writing that on the MIDI piano roll is slobber proof. But if you take music written that way and view it as sheet music, it is nearly impossible to read because of the many ties that don't even necessarily begin or even end in the same place as the others. It all ends up looking like layered sheets of snow sliding off your roof. A ribbon of differently-sagging tie-lines making it not only unsightly, but impossible to read. Why sheet music if you have MIDI already? Because as someone who records, I use MIDI for writing, but I don't expect other musicians to follow my piano rolls when recording guitar, bass, vocals, etc. and I wouldn't want them ever to sound as robotic as MIDI tends to be. Drums however - that's one area where MIDI piano roll to sheet music actually works flawlessly, even if the notes don't have the usual x| appearance.
- flipcoder 6y ago
  Does your midi software properly translate drum rudiments that have specific notation symbols? That would be quite impressive actually. Drumming has a lot of technique-specific and informal stylistic things that would be hard to translate.
amelius 6y ago
Perhaps they used MIDI only for reference during development. In an actual application they wouldn't need the MIDI. Just my guess.
- pishpash 6y ago
  If anyone bothered to click the link, it's not MIDI but MIDI-generated audio (title should be changed). They used MIDI audio because likely it simplified the problem along a number of dimensions: rhythm and timbre are fixed, and there is no noise. It also means it's cheating and it will not work on acoustic performances, but it's a good start. I mean if you can't solve the problem for MIDI audio you will never solve it for anything else.

adenadel 6y ago

Intuitively (to me), it feels like a clever variation of something like Needleman-Wunsch could pull this off without using machine learning (as long as you have the right encodings). Can someone enlighten me?

klodolph 6y ago
Sheet music is a sequence of images. Maybe you thought sheet music was more like a text file?
- adenadel 6y ago
  I should have been more clear. This is what I meant be "correct encodings". I would be surprised (but am welcome to being wrong) if a large amount of sheet music hadn't been encoded somehow.
  - klodolph 6y ago
    “Digitized” just means PDF, at best.
  - mjburgess 6y ago
    Sheet music isn't mathematical, ie., there isn't a numerical "encoding". It's has a fair amount of natural semantics (eg., "play like a fantasy"), There is only an embedding or re-presentation in some less informative form. ie., you have to choose some partial measurement system and apply this to the images of scores to produce a numeric representation. This is, in part, why a midi-file is always going to sound worse than an orchestra operating from sheet music. The musicians understand the intensions of the composer.
yorwba 6y ago
The sheet music is input as an image, which is not the right encoding for comparison with MIDI, so you need some kind of image recognition in the pipeline. I do think that they could replace the recurrent part of their network by dynamic programming and only use machine learning for feature extraction.

swiley 6y ago

Does this really need AI? Couldn't you just take the maximum of the fourier transform of the rendering multiplied with the audio? Maybe across multiple dimensions.

klodolph 6y ago
This kind of problem has been extensively studied for years and it turns out that it is not that easy. The fundamental is not the strongest harmonic, the harmonics don't line up, what happens when you play multiple notes at the same time, etc. It is a big stinking mess.
- iainctduncan 6y ago
  And that's before even separating out by instrument! It's really quite incredible what goes on between the ear and the brain for this kind of thing.
  - pishpash 6y ago
    Correlative methods are the wrong model by themselves but could be one part of the right model. Cognition happens by correlation, but also by denoising, by adaptive filtering, by interference cancellation, even by predictive generation. Listening is not a passive process.
- creato 6y ago
  There's a game called Rocksmith that has been on the market for many years that basically does this in real time on live guitar playing with far more than enough accuracy to make the game fun. And that is doing it basically per-note! This task (aligning sheet music with MIDI) would allow for global optimization approaches which should be far easier and more likely to work.
  - kevingadd 6y ago
    Electric guitar going through a clean (ish) input is a pretty narrowly scoped problem that won't necessarily generalize

deleted 6y ago

[deleted]

gnrlst 6y ago

I'm going through various books by Arnold Schoenberg. He adds lots of tiny examples that I dont have the skill to play nor the ear to "listen" to in my head. HN: Is there a tool/service/app that allows me to point at a piece of sheet music and it just plays it? I've tried a couple apps but they don't work well.

natn 6y ago

What I don't get is why the emphasis on sheet music as an image? I get that mapping from an auditory stream onto a score is an interesting problem, but isn't it naturally decoupled from the visual processing problem of reading and segmenting images of sheet music? Wouldn't it make more sense to deal primarily with a digital intermediate encoding of the score? What am I missing?

bane 6y ago

I wonder how this performs on Black MIDI https://www.youtube.com/watch?v=tfrIXiemYqo&ab_channel=Gingeas https://www.youtube.com/watch?v=tfrIXiemYqo&ab_channel=Ginge...