4 ms·

I used to work in an audio processing research center back in 2003, and colleagues next to me were able to isolate each instrument in a stereo mix live using th

by bsaul 8y ago

I used to work in an audio processing research center back in 2003, and colleagues next to me were able to isolate each instrument in a stereo mix live using the fact that they were "placed" on different spot in the stereo plane. Don't ask me how they did that, it was close to magic to me at that time, but i'm sure it wasn't neural networks. Although it probably involved convolution, as it is the main tool for producing audio filters. If anyone has more info on the fundamental differences of the neural network approach compared to the "traditional" one, i'd be thankful.

hooloovoo_zoo 8y ago
Historically, one approach has been independent component analysis (https://en.wikipedia.org/wiki/Independent_component_analysis https://en.wikipedia.org/wiki/Independent_component_analysis).
tomc1985 8y ago
There's a trick you can do to isolate vocals from some music by essentially flipping one of the stereo channels and combining their waveforms. All the stereo data cancels out and you're left with anything not panned hard center. Recombine that with the original stereo file converted to mono and you then get the vocals, usually a bunch of cruft from the reverb, and anything else panned hard center
- bsmith 8y ago
  This works especially well for pop tunes, where the vocal track is by far the most prominent and is almost universally panned dead-center in the mix.
- urvader 8y ago
  Most smartphones shift phases on the left/right channels to make it sound better in headphones (simulating bigger room), this means you can combine the two channels (by connecting the positive wires or just wiggle a little in the headphone jack half way in) to extract the vocal track. Not as fancy as a CNN but I’m guessing it would be better to do something with the stereo information in preprocessing before training.
- skykooler 8y ago
  Interestingly, with a Nexus 5 and a certain pair of earbuds, this seems to happen automatically (minus the last step) - with most songs, I can hear the backing tracks, but no vocals. The earbuds are passive components, so I assume there's something going on electrically that's combining the stereo channels somehow.
- tobr 8y ago
  Removing the center is possible, but how exactly would you "recombine" it with the original to keep the center? Correct me if I'm wrong, but I don't think the math works out like that. Say we have mixed three sources, let's call them L (panned hard left), C (center) and R (hard right). Then the left channel has +L+C, and the right channel has +R+C. Now we phase invert one of them, say the right channel, and combine them. The new mono file is +L+C-R-C. +C-C cancels out and we're left with +L-R. Since +R and -R essentially sounds the same, it sounds like if we had originally done a mono mix of L and R (+L+R). But we can't combine this with a straight mono conversion (+L+C+R+C) in any way that will remove both L and R. All we can do is reproduce +L+C or +R+C.
  - tomc1985 8y ago
    A stereo audio file only has two channels, mixed from any number of sources: L+R. This described process essentially converts the track (we'll call X) to mid/side (M/S) form, which is used by some microphones and other outboard equipment. We're exploiting properties of both forms of the audio to produce our isolated center. Also if you have the removed data all you have to do is invert it and sum it against hte original material and it will cancel out I described it slightly incorrectly in the original post but I have used this process to extract vocals for remixing The first inversion produces one channel containing nothing but stereo data, as you're summing L+Ri to produce S -- the center channel data (technically, anything equivalent in both stereo channels) cancelled itself out. (Note that if you combine L+Ri you should get a zero-ed out waveform) The second inversion, produced by combining the first inversion (Ri) and the original source converted to mono (L+R), produces the isolated center channel (M). It works because you are essentially only cancelling out stereo data you generated in the first inversion If you don't believe me, load up a copy of Audacity or Sound Forge and try it. (Note the music needs to be uncompressed) One track this works with is the original mix of "Day n Nite" by "Kid Cudi" if you can find a WAV copy somewhere. It doesn't work with a lot of music So... L, R = stereo_split(X) Ri = inverse(R) S = L+Ri Si = inverse(S) M = Si+(L+R) Note: L+Ri should result in empty waveform
    - tobr 8y ago
      Trying to follow this, but I’m lost at a few points. If you say you have a file where +L-R = 0, then L = R? That’s just a mono file? L/R stereo and M/S stereo conversion works like this. M = L+R, S = L-R (or S = R-L, either one is fine). If you expand out your expression M=Si+L+R, where Si=-(L-R)=-L+R, you have M=-L+R+L+R, which means you are saying M=+R+R. This doesn’t seem right. If you just want an actual M, that’s just a mono conversion of L+R. For sure, if the recording is M/S, the two sides cancel out and you will extract the mid channel, which would often contain song - but there’s no reason to do all the repeated inversion and mixing to do it!
bsmith 8y ago
I'm a fairly n00b-level hobbyist music producer, but I can take a stab. I like thinking of the "stereo image" of a track as having 3 dimensions in physical space. If you imagine looking forward at a stereo system, left to right is the "pan" (how much of a signal is originating from the left vs. right speakers, with 50% each being dead center), volume of each track (or instrument) is how distant (or close) the sound is (you can imagine an individual instrument moving forwards toward you if louder, and vice versa), and the frequency (or pitch) is how high the track is (with lower pitch or frequency being at the floor). When a mixing engineer "mixes" a track, each track (or instrument) tends to be: 1) adjusted to an appropriate volume relative to other tracks (forward/backward), panned or spread left/right (usually more stereo "width" on the higher frequency sounds), and "equalized" to narrow and manipulate the band of frequencies coming through to the master mix from that track (it's common to cut out some of the harmonics either side of the fundamental of one instrument to leave "space" for another instrument competing for the same frequency range). So now, even with a final, mastered track, we could apply various filters to de-mix fairly easily if the mixing engineer has left a good amount of "space" in between the elements in this stereo image. If our bass is always under 150 Hz and every other element is above 200 Hz, a simple low pass will grab the bass (and likely also the kick drum, if there is one). But we could also use a "gate" to only allow a signal of a certain magnitude (volume) through to isolate that kick hit, or the inverse to exclude the kick and just get the bass sound. A band-pass could do a decent job on a guitar or vocal track, but will also grab background noise from other tracks sharing the same frequency range. More complicated techniques could be used to isolate things that have been panned left or right of center based on comparison between the left and right stereo channels of the mix. These techniques won't be as universally useful as the approach in this article, but for certain tracks or sections of tracks, can give very good results very quickly. More tracks and effects like reverb and distortion make all of this more difficult to do with simpler techniques.
- vonseel 8y ago
  I think you’re overestimating how independent each instruments section of the frequency spectrum is, even after mixing engineers cutting EQ to make places like you said. The reality is, even instruments like kick drum or bass guitar have significant tonal content above 200hz, much of it overlapping with the guitar and vocal ranges.
meatsock 8y ago
you can compare the L and R channels to each other to separate signals out of a stereo field. https://cycling74.com/forums/separating-stereo-segments-possible https://cycling74.com/forums/separating-stereo-segments-poss...
derekp7 8y ago
Back when I was a teen, I used to strip out the vocal tracks on stereo music, by disconnecting the speaker ground wires from the back of the amplifier and connecting them to each other (so each speaker would ground against the other one). Since the vocal was "center", it had the same waveform on both speakers, so the speakers couldn't make a sound if they couldn't dump the ground to something. And the instrumentals weren't distorted too much. At least it worked good enough for a cheap Karaoke setup.
- robbrown451 8y ago
  I discovered this in the 80s when wires in to my car stereo speakers came loose and suddenly the vocal or lead instrument was missing from the music. Really puzzled me for a bit til I found the problem.
- redsky17 8y ago
  You can generally do this pretty closely digitally by taking the left and right channels of a stereo track and inverting the phase on one of them. Since the vocals are usually panned center (as you said), the inverted phase ends up destructively cancelling them. Depending on how the instrumentation was mixed, usually the instruments are left pretty in-tact.
  - codedokode 8y ago
    Usually the track sounds awful. For example, the bass can go away with the vocals, instruments' timbre change etc.
    - vonseel 8y ago
      Makes sense since panning bass would be an odd thing to do.
- heywire 8y ago
  I discovered something similar by accident, only with the line audio cables between by tape/CD and the amplifier. I would pull the plugs out just enough so the tip made connection but not the ring. It would give the same effect.
nabla9 8y ago
>using the fact that they were "placed" on different spot in the stereo plane. From the description it looks like they used FastICA. https://en.wikipedia.org/wiki/FastICA https://en.wikipedia.org/wiki/FastICA The traditional is faster and less recourse intensive. This approach is just showing that it can be done at this point.
cannam 8y ago
Almost certainly something like "Real-time Sound Source Separation: Azimuth Discrimination and Resynthesis" (Barry et al, https://arrow.dit.ie/argcon/35/ https://arrow.dit.ie/argcon/35/) This is a more sophisticated generalisation of the idea of inverting one channel and averaging in order to isolate the centre-panned vocal (or remove it for karaoke purposes). It works well with mixes in which stereo placement is entirely by pan pot, adjusting the left/right volume levels of each instrument individually in order to place it on the stereo image. It doesn't work so well with real room recordings, where stereo placement is determined also by timing information (read e.g. https://www.audiocheck.net/audiotests_stereophonicsound.php https://www.audiocheck.net/audiotests_stereophonicsound.php) This is a better-specified problem than monoaural source separation, which I think is what the original article here is about.

by bsaul 8y ago

hooloovoo_zoo 8y ago

Historically, one approach has been independent component analysis (https://en.wikipedia.org/wiki/Independent_component_analysis https://en.wikipedia.org/wiki/Independent_component_analysis).

tomc1985 8y ago

There's a trick you can do to isolate vocals from some music by essentially flipping one of the stereo channels and combining their waveforms. All the stereo data cancels out and you're left with anything not panned hard center. Recombine that with the original stereo file converted to mono and you then get the vocals, usually a bunch of cruft from the reverb, and anything else panned hard center

bsmith 8y ago
This works especially well for pop tunes, where the vocal track is by far the most prominent and is almost universally panned dead-center in the mix.
urvader 8y ago
Most smartphones shift phases on the left/right channels to make it sound better in headphones (simulating bigger room), this means you can combine the two channels (by connecting the positive wires or just wiggle a little in the headphone jack half way in) to extract the vocal track. Not as fancy as a CNN but I’m guessing it would be better to do something with the stereo information in preprocessing before training.
skykooler 8y ago
Interestingly, with a Nexus 5 and a certain pair of earbuds, this seems to happen automatically (minus the last step) - with most songs, I can hear the backing tracks, but no vocals. The earbuds are passive components, so I assume there's something going on electrically that's combining the stereo channels somehow.
tobr 8y ago
Removing the center is possible, but how exactly would you "recombine" it with the original to keep the center? Correct me if I'm wrong, but I don't think the math works out like that. Say we have mixed three sources, let's call them L (panned hard left), C (center) and R (hard right). Then the left channel has +L+C, and the right channel has +R+C. Now we phase invert one of them, say the right channel, and combine them. The new mono file is +L+C-R-C. +C-C cancels out and we're left with +L-R. Since +R and -R essentially sounds the same, it sounds like if we had originally done a mono mix of L and R (+L+R). But we can't combine this with a straight mono conversion (+L+C+R+C) in any way that will remove both L and R. All we can do is reproduce +L+C or +R+C.
- tomc1985 8y ago
  A stereo audio file only has two channels, mixed from any number of sources: L+R. This described process essentially converts the track (we'll call X) to mid/side (M/S) form, which is used by some microphones and other outboard equipment. We're exploiting properties of both forms of the audio to produce our isolated center. Also if you have the removed data all you have to do is invert it and sum it against hte original material and it will cancel out I described it slightly incorrectly in the original post but I have used this process to extract vocals for remixing The first inversion produces one channel containing nothing but stereo data, as you're summing L+Ri to produce S -- the center channel data (technically, anything equivalent in both stereo channels) cancelled itself out. (Note that if you combine L+Ri you should get a zero-ed out waveform) The second inversion, produced by combining the first inversion (Ri) and the original source converted to mono (L+R), produces the isolated center channel (M). It works because you are essentially only cancelling out stereo data you generated in the first inversion If you don't believe me, load up a copy of Audacity or Sound Forge and try it. (Note the music needs to be uncompressed) One track this works with is the original mix of "Day n Nite" by "Kid Cudi" if you can find a WAV copy somewhere. It doesn't work with a lot of music So... L, R = stereo_split(X) Ri = inverse(R) S = L+Ri Si = inverse(S) M = Si+(L+R) Note: L+Ri should result in empty waveform
  - tobr 8y ago
    Trying to follow this, but I’m lost at a few points. If you say you have a file where +L-R = 0, then L = R? That’s just a mono file? L/R stereo and M/S stereo conversion works like this. M = L+R, S = L-R (or S = R-L, either one is fine). If you expand out your expression M=Si+L+R, where Si=-(L-R)=-L+R, you have M=-L+R+L+R, which means you are saying M=+R+R. This doesn’t seem right. If you just want an actual M, that’s just a mono conversion of L+R. For sure, if the recording is M/S, the two sides cancel out and you will extract the mid channel, which would often contain song - but there’s no reason to do all the repeated inversion and mixing to do it!

bsmith 8y ago

I'm a fairly n00b-level hobbyist music producer, but I can take a stab. I like thinking of the "stereo image" of a track as having 3 dimensions in physical space. If you imagine looking forward at a stereo system, left to right is the "pan" (how much of a signal is originating from the left vs. right speakers, with 50% each being dead center), volume of each track (or instrument) is how distant (or close) the sound is (you can imagine an individual instrument moving forwards toward you if louder, and vice versa), and the frequency (or pitch) is how high the track is (with lower pitch or frequency being at the floor). When a mixing engineer "mixes" a track, each track (or instrument) tends to be: 1) adjusted to an appropriate volume relative to other tracks (forward/backward), panned or spread left/right (usually more stereo "width" on the higher frequency sounds), and "equalized" to narrow and manipulate the band of frequencies coming through to the master mix from that track (it's common to cut out some of the harmonics either side of the fundamental of one instrument to leave "space" for another instrument competing for the same frequency range). So now, even with a final, mastered track, we could apply various filters to de-mix fairly easily if the mixing engineer has left a good amount of "space" in between the elements in this stereo image. If our bass is always under 150 Hz and every other element is above 200 Hz, a simple low pass will grab the bass (and likely also the kick drum, if there is one). But we could also use a "gate" to only allow a signal of a certain magnitude (volume) through to isolate that kick hit, or the inverse to exclude the kick and just get the bass sound. A band-pass could do a decent job on a guitar or vocal track, but will also grab background noise from other tracks sharing the same frequency range. More complicated techniques could be used to isolate things that have been panned left or right of center based on comparison between the left and right stereo channels of the mix. These techniques won't be as universally useful as the approach in this article, but for certain tracks or sections of tracks, can give very good results very quickly. More tracks and effects like reverb and distortion make all of this more difficult to do with simpler techniques.

vonseel 8y ago
I think you’re overestimating how independent each instruments section of the frequency spectrum is, even after mixing engineers cutting EQ to make places like you said. The reality is, even instruments like kick drum or bass guitar have significant tonal content above 200hz, much of it overlapping with the guitar and vocal ranges.

meatsock 8y ago

you can compare the L and R channels to each other to separate signals out of a stereo field. https://cycling74.com/forums/separating-stereo-segments-possible https://cycling74.com/forums/separating-stereo-segments-poss...

derekp7 8y ago

Back when I was a teen, I used to strip out the vocal tracks on stereo music, by disconnecting the speaker ground wires from the back of the amplifier and connecting them to each other (so each speaker would ground against the other one). Since the vocal was "center", it had the same waveform on both speakers, so the speakers couldn't make a sound if they couldn't dump the ground to something. And the instrumentals weren't distorted too much. At least it worked good enough for a cheap Karaoke setup.

robbrown451 8y ago
I discovered this in the 80s when wires in to my car stereo speakers came loose and suddenly the vocal or lead instrument was missing from the music. Really puzzled me for a bit til I found the problem.
redsky17 8y ago
You can generally do this pretty closely digitally by taking the left and right channels of a stereo track and inverting the phase on one of them. Since the vocals are usually panned center (as you said), the inverted phase ends up destructively cancelling them. Depending on how the instrumentation was mixed, usually the instruments are left pretty in-tact.
- codedokode 8y ago
  Usually the track sounds awful. For example, the bass can go away with the vocals, instruments' timbre change etc.
  - vonseel 8y ago
    Makes sense since panning bass would be an odd thing to do.
heywire 8y ago
I discovered something similar by accident, only with the line audio cables between by tape/CD and the amplifier. I would pull the plugs out just enough so the tip made connection but not the ring. It would give the same effect.

nabla9 8y ago

>using the fact that they were "placed" on different spot in the stereo plane. From the description it looks like they used FastICA. https://en.wikipedia.org/wiki/FastICA https://en.wikipedia.org/wiki/FastICA The traditional is faster and less recourse intensive. This approach is just showing that it can be done at this point.

cannam 8y ago

Almost certainly something like "Real-time Sound Source Separation: Azimuth Discrimination and Resynthesis" (Barry et al, https://arrow.dit.ie/argcon/35/ https://arrow.dit.ie/argcon/35/) This is a more sophisticated generalisation of the idea of inverting one channel and averaging in order to isolate the centre-panned vocal (or remove it for karaoke purposes). It works well with mixes in which stereo placement is entirely by pan pot, adjusting the left/right volume levels of each instrument individually in order to place it on the stereo image. It doesn't work so well with real room recordings, where stereo placement is determined also by timing information (read e.g. https://www.audiocheck.net/audiotests_stereophonicsound.php https://www.audiocheck.net/audiotests_stereophonicsound.php) This is a better-specified problem than monoaural source separation, which I think is what the original article here is about.