3 ms·

This was enjoyable. I wonder if software exists that is aware of the overtones of each particular note of a given instrument. Theoretically, this would be a mo

This was enjoyable. I wonder if software exists that is aware of the overtones of each particular note of a given instrument. Theoretically, this would be a more precise way to pull out a chord from a recording without having to manually compensate for them later. Standard Fourier analysis is based on sine waves which aren't the best model for complex tones like that of a piano or a 12-string guitar. Mathematically, the best approach I'm aware of in the long run would be to study the harmonic spectrum of the exact instrument as closely as possible, and then perform inner products against the spectrum of the mystery sound being analyzed. There's no guarantee that the note-vectors would be orthogonal (probably not), so the problem becomes more interesting in that you might want to perform a sparsity maximization on the resulting output - that is, look for the least number of notes capable of matching the recording within epsilon error in sample (or spectrum) space. Sparsity maximization is NP-complete, so a good way to approximate this is to actually perform L1-minimization using linear programming. Another approach might be to look for a balance between the absolute error term and the sparsity term.

TTPrograms 12y ago
If you know the number of notes being played you can solve in polynomial time. You could run it with a few variations in the assumed number of notes. The harmonic spectrum of each instrument will be extremely difficult to get, though, as each note can have a different waveform and will be affected by unknown non-linear effects in the instrument itself as well as in mixing (i.e. compression or distortion).
- kefka 12y ago
  Not really. First, you can simulate overtones by where you strike a guitar. Secondly, we don't know the model of the tube amps they were using. Tube amps do some really funky mixing, which can exasperate FFTs by giving weird/wrong results.
  - TTPrograms 12y ago
    >First, you can simulate overtones by where you strike a guitar. If you know the make of the guitar and the strings and where it was picked each time and the location of each of the tone and volume knobs and the position of the pickup switches. >Tube amps do some really funky mixing, which can exasperate FFTs by giving weird/wrong results. >unknown non-linear effects...distortion
jay-anderson 12y ago
Agreed -- Just looking for individual spikes in the harmonic spectrum is naive. However, I think even if you have a good idea of which instrument it is there are other factors involved which could make matching a generic instrument model more difficult (e.g. the player, the playing space, instrument variations, etc.). For instance with the french horn (it's what I know) there are sometimes drastic differences between players and instruments -- the most stark being possibly the vienna horn vs. the usual double horn.
pierrec 12y ago
Yes, software like this exists. Using multiple overtones to reconstruct a single note is pretty basic in this field, if you search for "wave to midi" software (for example) you'll find a few that do this. The most advanced right now might be Melodyne [1] (used in the article), which also allows you to pitch-shift the detected notes (and all their partials), without affecting the rest of the sound. However it's quite proprietary, and I don't think the details of their algorithms are available, but it does a good job on a variety of instruments. The polyphonic detection can never be perfect, of course, especially with layered organic instruments such as voice and winds, it becomes impossible to determine what note a particular partial belongs to. [1]: http://en.wikipedia.org/wiki/Celemony http://en.wikipedia.org/wiki/Celemony

TTPrograms 12y ago

If you know the number of notes being played you can solve in polynomial time. You could run it with a few variations in the assumed number of notes. The harmonic spectrum of each instrument will be extremely difficult to get, though, as each note can have a different waveform and will be affected by unknown non-linear effects in the instrument itself as well as in mixing (i.e. compression or distortion).

kefka 12y ago
Not really. First, you can simulate overtones by where you strike a guitar. Secondly, we don't know the model of the tube amps they were using. Tube amps do some really funky mixing, which can exasperate FFTs by giving weird/wrong results.
- TTPrograms 12y ago
  >First, you can simulate overtones by where you strike a guitar. If you know the make of the guitar and the strings and where it was picked each time and the location of each of the tone and volume knobs and the position of the pickup switches. >Tube amps do some really funky mixing, which can exasperate FFTs by giving weird/wrong results. >unknown non-linear effects...distortion

jay-anderson 12y ago

Agreed -- Just looking for individual spikes in the harmonic spectrum is naive. However, I think even if you have a good idea of which instrument it is there are other factors involved which could make matching a generic instrument model more difficult (e.g. the player, the playing space, instrument variations, etc.). For instance with the french horn (it's what I know) there are sometimes drastic differences between players and instruments -- the most stark being possibly the vienna horn vs. the usual double horn.

pierrec 12y ago

Yes, software like this exists. Using multiple overtones to reconstruct a single note is pretty basic in this field, if you search for "wave to midi" software (for example) you'll find a few that do this. The most advanced right now might be Melodyne [1] (used in the article), which also allows you to pitch-shift the detected notes (and all their partials), without affecting the rest of the sound. However it's quite proprietary, and I don't think the details of their algorithms are available, but it does a good job on a variety of instruments. The polyphonic detection can never be perfect, of course, especially with layered organic instruments such as voice and winds, it becomes impossible to determine what note a particular partial belongs to. [1]: http://en.wikipedia.org/wiki/Celemony http://en.wikipedia.org/wiki/Celemony