7 ms·

You would think getting the notes and converting them to chords, and tablature, would be one of those "exactly suited for neural nets" type of problem. If you

You would think getting the notes and converting them to chords, and tablature, would be one of those "exactly suited for neural nets" type of problem. If you want the chords to "Here comes the sun," you can find dozens of hits, but try something slightly obscure and they are hard to come by. (People with great ears have no idea what I am talking about.)

cnity 5y ago
I've worked on this problem for some time on a personal project, and I'm pretty convinced you can basically solve this problem without deep learning or AI techniques, and instead use non-negative matrix factorization[0] as a bank of note templates (from their spectrograms). I have a fairly well working proof of concept and the approach is supported by the literature. [0]: https://en.wikipedia.org/wiki/Non-negative_matrix_factorization https://en.wikipedia.org/wiki/Non-negative_matrix_factorizat... _edit_ That said, you'd probably need something more hard-core for extraction from an actual track, so you're probably right.
- anonymouse008 5y ago
  This is the right approach for song recommendation too — try your approach there and see what happens. If you want help on the business side, reach out.
  - cnity 5y ago
    That is very kind of you to offer. I don't know if I'd be able to find the time to dedicate to having this function as an actual product though!
    - anonymouse008 5y ago
      You're already leaps and bounds into a good partnership - realistic expectations. Consider it again when you get downtime, no rush over here, more fun.
  - rodw 5y ago
    Can you expand on what you mean by "song recommendations" in this context? Do you mean recommendations like "if you like X you might also like Y"? Assuming the answer is "yes", I'm not sure if I follow the leap from "non-negative matrix factorization as a bank of note templates (from their spectrograms)" to "song recommendations". Very loosely speaking my (not wholly uninformed) interpretation of the "note templates" bit is sorta analogous to DFFT analysis with frequency bins centered around the "regular" notes of the chromatic scale - i.e., the cells of the matrix represent signal strength for the frequencies that correspond to conventional western notes (e.g. integer-valued midi note numbers) or (in aggregated form) octave-independent pitch-classes rather than arbitrary frequencies (that might fall between two conventional notes). It's a very useful representation of the component frequencies that appear in conventional music but at the end of the day it's more or less "here are the notes (or pitch classes) active for this given beat". That is, I can imagine the role this information - essentially the musical score itself, or at least the pitch-specific dimension of that score - could play in a song-recommendation engine, but I'm curious how/why that specific spectrogram-template-based representation is significant. Are you suggesting that that specific representation could be applied to song recommendations in a way that similar polyphonic-pitch-per-beat information derived from some other algorithm (FFT analysis for a hypothetical example) could not? Or maybe I've misinterpreted your comment entirely?
    - cnity 5y ago
      More likely, they mean using non-negative matrix factorization but with a bank of feature vectors instead of note templates. NNMF can be used in a wide variety of domains because it essentially encodes the problem of "this thing is a bit like this thing, a bit like this thing, and a bit like this other thing". If instead of numbers representing intensity at different frequencies (as in the spectrograms), the numbers in each vector of the template bank represent other features (such as listener overlap with other artists/songs, or genre representation across multiple continuous "color" axes) then you can recommend music to a listener based on the similarity to songs in their library to those in the template bank.
      - rodw 5y ago
- notfed 5y ago
  As an amateur musician I see this as a missing holy grail killer app. I'd love to have it just to pull the chords out of some of my own old recordings that I can't figure out how to repeat. If anyone knows of any apps (even prototypes) that can do this, please provide links.
  - cnity 5y ago
    I may have to blow the dust off of my prototype.
    - moehm 5y ago
      If you do, make sure to make it a Show HN to let us know!
  - cannam 5y ago
    There are various commercial offerings that do this, including Chordify and Capo. Alternatively if you want a flexible free (GPLv2) option with source code, consider the Chordino plugin (http://www.isophonics.net/nnls-chroma http://www.isophonics.net/nnls-chroma, C++ code at https://github.com/c4dm/nnls-chroma https://github.com/c4dm/nnls-chroma) which you can run in Sonic Visualiser (https://www.sonicvisualiser.org/ https://www.sonicvisualiser.org/), or indeed in Audacity. There is a reference describing the Chordino method in the page linked above, but roughly it's not too far from the description of the parent poster - a non-negative least-squares method produces a frame-by-frame semitone-scaled decomposition which is then matched against templates and turned into a chord sequence using a hidden Markov model. Some sort of intelligent smoothing like this is certainly needed, the raw template matching is not especially useful on its own. This type of method is now routinely outperformed by neural networks (see e.g. these MIREX evaluation results from 2018 which compare Chordino with a few other academic methods https://www.music-ir.org/mirex/wiki/2018:Audio_Chord_Estimation_Results https://www.music-ir.org/mirex/wiki/2018:Audio_Chord_Estimat...) but I would suggest that it's still good enough to be useful - and I encourage the parent to continue their work, as it's an interesting area to explore.
    - notfed 5y ago
- rodw 5y ago
  I've also spent a fair bit of time on this topic and for what it's worth I agree with you. It is a harder problem than the monophonic case (and more sensitive to problems like noise under real-world conditions) but you don't strictly need deep learning or AI techniques to solve it. I mean, computational complexity aside it seems like at least hypothetically you could even just apply basic auto-correlation-style logic to detect the period of the combined wave much like you do in the monophonic case (assuming the chord is sustained for long enough to actually capture that full period, which of course it won't be in the general case). There's nothing magical about a neural-net or other deep-learning-style solution to this problem - at the end of the day that's just an approximation of a formula that could in theory be derived through more direct means anyway. And (as far as I know) there's no reason to believe the polyphonic case is fundamentally resistant to more traditional techniques. And as implied by your comment, the problem is made easier (or at least less resource-intensive) in practice than it is in the abstract: we're mostly interested in audio that's comprised of actual notes from the chromatic scale (rather than a combination of arbitrary frequencies). There's only ~140 or so component frequencies we really need to consider in practice. (Not to mention the semi-predictable repetition/progression patterns you're likely to encounter in most conventional songs. That's inadequate by itself but certainly a good way to error correct, fill in gaps, resolve ambiguous cases, etc.) But that said, it does seem like polyphonic pitch detection is a problem that responds really well to machine-learning techniques. In my experience, even a fairly simplistic ANN (e.g., no hidden layers, ~1k to ~10k weights depending upon how the inputs/outputs are modeled) - when seeded with a little bit of domain-specific knowledge - can very quickly learn to perform reliable polyphonic pitch detection under real-world conditions. To be fair, I haven't quite put my money where my mouth is on this topic (yet): I develop software that includes this sort of functionality and the current production version uses more conventional (or at least direct) analysis rather than so called "deep learning" techniques for polyphonic pitch detection. There are pros and cons to either approach, but I can definitely see why some find the deep learning solution attractive. There's probably some degree of magical thinking involved (i.e., "AI will solve this pattern recognition problem that's too hard for me to work out from first principles"), but it also seems to work really well in this case. For what it's worth I think you've got the right general idea, or at least (based on your brief description) I think I arrived at a solution that's based on some similar concepts and found it fairly effective (beyond the proof-of-concept phase). And as you noted there are related concepts discussed in some of the published academic research. I'd love to hear a little more about your approach if you're willing and able to share any more details. (Noting that at least part of my interest in that topic is selfish, of course.)
astrea 5y ago
Thus propagating the NN "If all you have is a hammer" trend. I don't quite know the nuance of music theory, but could you not get away with traditional Fourier analysis? You just need to decompose the song into its constituent frequency "bins" right?
- cnity 5y ago
  The trouble with analysing "notes" in a composition (or even just with a polyphonic instrument) is the pesky harmonics. The timbre of different instruments produce harmonics that happen to be the fundamental frequencies of other notes.
  - etskinner 5y ago
    Would it be possible to eke out the harmonics by saying (for 'x' type of guitar tuned in 'y' way with 'z' effect, harmonics look like this). Like if the fundamental frequency is f, then the whole note looks like 0db at f and -5db at 2f and -10db at 3f or something. Then, when you're looking at the frequency domain, you start from the lower notes and say "hmm, looks like there's a fundamental at f, are the expected harmonics there"? If yes, that's the note, if no, it's something else.
    - cnity 5y ago
      This is basically how you'd do it with non-negative matrix factorization. You take the spectrogram of a bunch of known notes for that type of guitar, and store them in a template library (just a vector of vectors, where each inner vector is a spectrogram of that note). Then NNMF determines how much of each template contributes to some given signal. The templates are the "harmonics look like this" thing you're talking about. It works pretty well. By the way, the reason it is easier to use NNMF than try to implement your suggestion as a heuristic is because there's much more overlap between the different notes than you might think, and (worst of all) the timbre of the note: 1. evolves over time! 2. depends on the velocity of the note (how hard the string was strum) 3. and the notes actually interact with each other. If you play an E, the A string will mildly reverberate too because of the shared harmonics
CardenB 5y ago
Ubisoft has a new version of rocksmith coming out and plans to build most of their library on this technology.
- bentcorner 5y ago
  Wait, really? I would think they would use some kind of canonical source for the tabs they've licensed. Maybe I expect too much.
  - colejohnson66 5y ago
    With OG Rocksmith (and 2014), the charts were supposedly all done by the folks at Ubisoft. They'd watch live videos and have multiple go-overs with each other to make it as accurate as possible. Besides, "official" tabs (at least historically) were usually not done by the actual artists, but some company that was just given the recordings and paid to figure it out.
  - CardenB 5y ago
    They have two modes: 1. Transcribed tabs. They own the transcription as they have done with their catalogue in existing Rocksmith games. I believe they claim they use some “AI assistance” here, but I could be wrong. At the end of the day, they pay a large number of musicians to chew through songs and write tabs. 2. Chord following. They use automated transcription to get the chords for a very large number of songs. This is a new mode in the newest game and is entirely automated, but limited.
- gramie 5y ago
  Ah yes, Rocksmith+ that requires a subscription to use, and will not work if you are not connected to the internet (i.e. if your subscription lapses, your purchase is worthless). It's now been delayed until 2022, I wonder if the feedback made them realize that while the board may love the idea of mandatory subscriptions, customers hate it.
dodgrile 5y ago
I don't think finding individual notes / the inherent notes within a chord is likely that difficult. The problem with this sort of thing is the nuance that's really involved - a player with a decent ear will be able to tell if the same A note is played on an open string, on the lower, fatter strings, or further up the neck. With chords, you also need to start considering voice leading (where you're specifically picking chord inversions for melody, which on guitar will affect how / where you play a chord) and how the mechanics fit together overall (I probably don't want to play an open C chord followed by something on the 11th fret, for example).
- omarhaneef 5y ago
  Tell me you have a good ear without telling me you have a good ear.
- tyingq 5y ago
  Aren't there a fair amount of songs where the guitar is in some non-standard tuning or played with a kapo, etc? I imagine a naive ML approach would spit out some tabs that would be difficult to play without that context.
- gerbilly 5y ago
  Yeah exactly what I was going to say, you have to consider the chord voicing, the strum pattern, and whether the sequence of chords makes sense for a human to comfortably play. Someone above mentioned: "Here comes the sun", which is a great example. First gotcha, it sort of sounds finger picked, but it's not it's flat picked. Another unusual thing about that song is that it's played w/ a capo on 7th fret. Now I'm sure some software could figure out the individual notes, but I wouldn't be surprised if it transcribed it as nonsense sequence of notes all over on the g, b and e strings, instead of arpeggiated chords as it really is played.
  - siltpotato 5y ago
    Maybe do away with tabbing and just give out sheet music.
- 5y ago

cnity 5y ago

I've worked on this problem for some time on a personal project, and I'm pretty convinced you can basically solve this problem without deep learning or AI techniques, and instead use non-negative matrix factorization[0] as a bank of note templates (from their spectrograms). I have a fairly well working proof of concept and the approach is supported by the literature. [0]: https://en.wikipedia.org/wiki/Non-negative_matrix_factorization https://en.wikipedia.org/wiki/Non-negative_matrix_factorizat... _edit_ That said, you'd probably need something more hard-core for extraction from an actual track, so you're probably right.

anonymouse008 5y ago
This is the right approach for song recommendation too — try your approach there and see what happens. If you want help on the business side, reach out.
- cnity 5y ago
  That is very kind of you to offer. I don't know if I'd be able to find the time to dedicate to having this function as an actual product though!
  - anonymouse008 5y ago
    You're already leaps and bounds into a good partnership - realistic expectations. Consider it again when you get downtime, no rush over here, more fun.
- rodw 5y ago
  Can you expand on what you mean by "song recommendations" in this context? Do you mean recommendations like "if you like X you might also like Y"? Assuming the answer is "yes", I'm not sure if I follow the leap from "non-negative matrix factorization as a bank of note templates (from their spectrograms)" to "song recommendations". Very loosely speaking my (not wholly uninformed) interpretation of the "note templates" bit is sorta analogous to DFFT analysis with frequency bins centered around the "regular" notes of the chromatic scale - i.e., the cells of the matrix represent signal strength for the frequencies that correspond to conventional western notes (e.g. integer-valued midi note numbers) or (in aggregated form) octave-independent pitch-classes rather than arbitrary frequencies (that might fall between two conventional notes). It's a very useful representation of the component frequencies that appear in conventional music but at the end of the day it's more or less "here are the notes (or pitch classes) active for this given beat". That is, I can imagine the role this information - essentially the musical score itself, or at least the pitch-specific dimension of that score - could play in a song-recommendation engine, but I'm curious how/why that specific spectrogram-template-based representation is significant. Are you suggesting that that specific representation could be applied to song recommendations in a way that similar polyphonic-pitch-per-beat information derived from some other algorithm (FFT analysis for a hypothetical example) could not? Or maybe I've misinterpreted your comment entirely?
  - cnity 5y ago
    More likely, they mean using non-negative matrix factorization but with a bank of feature vectors instead of note templates. NNMF can be used in a wide variety of domains because it essentially encodes the problem of "this thing is a bit like this thing, a bit like this thing, and a bit like this other thing". If instead of numbers representing intensity at different frequencies (as in the spectrograms), the numbers in each vector of the template bank represent other features (such as listener overlap with other artists/songs, or genre representation across multiple continuous "color" axes) then you can recommend music to a listener based on the similarity to songs in their library to those in the template bank.
    - rodw 5y ago
notfed 5y ago
As an amateur musician I see this as a missing holy grail killer app. I'd love to have it just to pull the chords out of some of my own old recordings that I can't figure out how to repeat. If anyone knows of any apps (even prototypes) that can do this, please provide links.
- cnity 5y ago
  I may have to blow the dust off of my prototype.
  - moehm 5y ago
    If you do, make sure to make it a Show HN to let us know!
- cannam 5y ago
  There are various commercial offerings that do this, including Chordify and Capo. Alternatively if you want a flexible free (GPLv2) option with source code, consider the Chordino plugin (http://www.isophonics.net/nnls-chroma http://www.isophonics.net/nnls-chroma, C++ code at https://github.com/c4dm/nnls-chroma https://github.com/c4dm/nnls-chroma) which you can run in Sonic Visualiser (https://www.sonicvisualiser.org/ https://www.sonicvisualiser.org/), or indeed in Audacity. There is a reference describing the Chordino method in the page linked above, but roughly it's not too far from the description of the parent poster - a non-negative least-squares method produces a frame-by-frame semitone-scaled decomposition which is then matched against templates and turned into a chord sequence using a hidden Markov model. Some sort of intelligent smoothing like this is certainly needed, the raw template matching is not especially useful on its own. This type of method is now routinely outperformed by neural networks (see e.g. these MIREX evaluation results from 2018 which compare Chordino with a few other academic methods https://www.music-ir.org/mirex/wiki/2018:Audio_Chord_Estimation_Results https://www.music-ir.org/mirex/wiki/2018:Audio_Chord_Estimat...) but I would suggest that it's still good enough to be useful - and I encourage the parent to continue their work, as it's an interesting area to explore.
  - notfed 5y ago
rodw 5y ago
I've also spent a fair bit of time on this topic and for what it's worth I agree with you. It is a harder problem than the monophonic case (and more sensitive to problems like noise under real-world conditions) but you don't strictly need deep learning or AI techniques to solve it. I mean, computational complexity aside it seems like at least hypothetically you could even just apply basic auto-correlation-style logic to detect the period of the combined wave much like you do in the monophonic case (assuming the chord is sustained for long enough to actually capture that full period, which of course it won't be in the general case). There's nothing magical about a neural-net or other deep-learning-style solution to this problem - at the end of the day that's just an approximation of a formula that could in theory be derived through more direct means anyway. And (as far as I know) there's no reason to believe the polyphonic case is fundamentally resistant to more traditional techniques. And as implied by your comment, the problem is made easier (or at least less resource-intensive) in practice than it is in the abstract: we're mostly interested in audio that's comprised of actual notes from the chromatic scale (rather than a combination of arbitrary frequencies). There's only ~140 or so component frequencies we really need to consider in practice. (Not to mention the semi-predictable repetition/progression patterns you're likely to encounter in most conventional songs. That's inadequate by itself but certainly a good way to error correct, fill in gaps, resolve ambiguous cases, etc.) But that said, it does seem like polyphonic pitch detection is a problem that responds really well to machine-learning techniques. In my experience, even a fairly simplistic ANN (e.g., no hidden layers, ~1k to ~10k weights depending upon how the inputs/outputs are modeled) - when seeded with a little bit of domain-specific knowledge - can very quickly learn to perform reliable polyphonic pitch detection under real-world conditions. To be fair, I haven't quite put my money where my mouth is on this topic (yet): I develop software that includes this sort of functionality and the current production version uses more conventional (or at least direct) analysis rather than so called "deep learning" techniques for polyphonic pitch detection. There are pros and cons to either approach, but I can definitely see why some find the deep learning solution attractive. There's probably some degree of magical thinking involved (i.e., "AI will solve this pattern recognition problem that's too hard for me to work out from first principles"), but it also seems to work really well in this case. For what it's worth I think you've got the right general idea, or at least (based on your brief description) I think I arrived at a solution that's based on some similar concepts and found it fairly effective (beyond the proof-of-concept phase). And as you noted there are related concepts discussed in some of the published academic research. I'd love to hear a little more about your approach if you're willing and able to share any more details. (Noting that at least part of my interest in that topic is selfish, of course.)

astrea 5y ago

Thus propagating the NN "If all you have is a hammer" trend. I don't quite know the nuance of music theory, but could you not get away with traditional Fourier analysis? You just need to decompose the song into its constituent frequency "bins" right?

cnity 5y ago
The trouble with analysing "notes" in a composition (or even just with a polyphonic instrument) is the pesky harmonics. The timbre of different instruments produce harmonics that happen to be the fundamental frequencies of other notes.
- etskinner 5y ago
  Would it be possible to eke out the harmonics by saying (for 'x' type of guitar tuned in 'y' way with 'z' effect, harmonics look like this). Like if the fundamental frequency is f, then the whole note looks like 0db at f and -5db at 2f and -10db at 3f or something. Then, when you're looking at the frequency domain, you start from the lower notes and say "hmm, looks like there's a fundamental at f, are the expected harmonics there"? If yes, that's the note, if no, it's something else.
  - cnity 5y ago
    This is basically how you'd do it with non-negative matrix factorization. You take the spectrogram of a bunch of known notes for that type of guitar, and store them in a template library (just a vector of vectors, where each inner vector is a spectrogram of that note). Then NNMF determines how much of each template contributes to some given signal. The templates are the "harmonics look like this" thing you're talking about. It works pretty well. By the way, the reason it is easier to use NNMF than try to implement your suggestion as a heuristic is because there's much more overlap between the different notes than you might think, and (worst of all) the timbre of the note: 1. evolves over time! 2. depends on the velocity of the note (how hard the string was strum) 3. and the notes actually interact with each other. If you play an E, the A string will mildly reverberate too because of the shared harmonics

CardenB 5y ago

Ubisoft has a new version of rocksmith coming out and plans to build most of their library on this technology.

bentcorner 5y ago
Wait, really? I would think they would use some kind of canonical source for the tabs they've licensed. Maybe I expect too much.
- colejohnson66 5y ago
  With OG Rocksmith (and 2014), the charts were supposedly all done by the folks at Ubisoft. They'd watch live videos and have multiple go-overs with each other to make it as accurate as possible. Besides, "official" tabs (at least historically) were usually not done by the actual artists, but some company that was just given the recordings and paid to figure it out.
- CardenB 5y ago
  They have two modes: 1. Transcribed tabs. They own the transcription as they have done with their catalogue in existing Rocksmith games. I believe they claim they use some “AI assistance” here, but I could be wrong. At the end of the day, they pay a large number of musicians to chew through songs and write tabs. 2. Chord following. They use automated transcription to get the chords for a very large number of songs. This is a new mode in the newest game and is entirely automated, but limited.
gramie 5y ago
Ah yes, Rocksmith+ that requires a subscription to use, and will not work if you are not connected to the internet (i.e. if your subscription lapses, your purchase is worthless). It's now been delayed until 2022, I wonder if the feedback made them realize that while the board may love the idea of mandatory subscriptions, customers hate it.

dodgrile 5y ago

I don't think finding individual notes / the inherent notes within a chord is likely that difficult. The problem with this sort of thing is the nuance that's really involved - a player with a decent ear will be able to tell if the same A note is played on an open string, on the lower, fatter strings, or further up the neck. With chords, you also need to start considering voice leading (where you're specifically picking chord inversions for melody, which on guitar will affect how / where you play a chord) and how the mechanics fit together overall (I probably don't want to play an open C chord followed by something on the 11th fret, for example).

omarhaneef 5y ago
Tell me you have a good ear without telling me you have a good ear.
tyingq 5y ago
Aren't there a fair amount of songs where the guitar is in some non-standard tuning or played with a kapo, etc? I imagine a naive ML approach would spit out some tabs that would be difficult to play without that context.
gerbilly 5y ago
Yeah exactly what I was going to say, you have to consider the chord voicing, the strum pattern, and whether the sequence of chords makes sense for a human to comfortably play. Someone above mentioned: "Here comes the sun", which is a great example. First gotcha, it sort of sounds finger picked, but it's not it's flat picked. Another unusual thing about that song is that it's played w/ a capo on 7th fret. Now I'm sure some software could figure out the individual notes, but I wouldn't be surprised if it transcribed it as nonsense sequence of notes all over on the g, b and e strings, instead of arpeggiated chords as it really is played.
- siltpotato 5y ago
  Maybe do away with tabbing and just give out sheet music.
5y ago