3 ms·

Does anyone know if this is related to the new iZotope RX 7 vocal isolation & stemming tools? It does seem to be talking about something similar, especially wh

Does anyone know if this is related to the new iZotope RX 7 vocal isolation & stemming tools? It does seem to be talking about something similar, especially when it mentions using the same technique to split a song into instrument stems. (Or to put it another way - there is commercial music software released in the last year that lets you do this yourself now.) https://www.youtube.com/watch?v=kEauVQv2Quc https://www.youtube.com/watch?v=kEauVQv2Quc https://www.izotope.com/en/products/repair-and-edit/rx/music.html https://www.izotope.com/en/products/repair-and-edit/rx/music...

pizza 8y ago
Going back further, X-Tracks did this ~5 years ago https://vimeo.com/107971872 https://vimeo.com/107971872 That said, I think a deep learning approach will likely do a lot better (and be a lot easier to develop, imo) Also, check out Google's Magenta project; it aims to use ML in various music / creativity projects. I personally plan on doing a project that will involve audio source separation as well as sample classification; a good trick for analyzing audio data is to convert it into images (maybe with some additional pre-transformations applied, such as passing it through an audio filter that exaggerates human-perceived properties of sound) and then just use your run-of-the-mill, bog-standard, state-of-the-art image classifiers on the resulting audio spectrogram with some well-chosen training/validation sets.
- Eli_P 8y ago
  That's interesting, are you going to make something like LabelImg[1]? I've been looking for something like that for audio, yet I'm not sure about treating audio as images. I've heard of this trick, but NN for audio better do work with RNN, GRU[2], maybe LSTM; and images are processed with CNN. [1] https://github.com/tzutalin/labelImg https://github.com/tzutalin/labelImg [2] https://en.wikipedia.org/wiki/Gated_recurrent_unit https://en.wikipedia.org/wiki/Gated_recurrent_unit
  - pizza 8y ago
    I was gonna do something involving about 3 different neural nets: a source separator: taking one audio stream as input and producing a set of audio streams as output. a segmentation regression neural net: takes an audio stream as input and returns start and stop timestamps of individual samples as output, or alternativey, just trimmed copies of the audio stream sample classifier: takes an audio stream and then returns “kick drum”, “snare drum”, “voice”, “guitar”, etc then the pipeline would be like source separator => segmenter => sample classifier Hopefully with this I would be able to decompose music into constituent parts, useful for remixing and other kinds of musique concrete I expect that the results with a deep pretrained generic image model + some tweaking with more niche training examples will be satisfactory, but if not it would be a good excuse to experiment with more traditionally seuqnece-oriented network architectures
- ambicapter 8y ago
  Do you have to convert it into an image? What is it about classifiers that require image input? I've always found it very cool that audio compression and image compression end up using similar frequency-space techniques sometimes.
  - pizza 8y ago
    You don't have to. It's just that there are a lot of good image classification network architectures that exist already so you can more or less grab a good network off-the-shelf, then give it more targeted training examples so that it performs well in your specific use-case. I guess it comes down to the fact that a lot of these signal processing techniques involve convolution.

pizza 8y ago

Going back further, X-Tracks did this ~5 years ago https://vimeo.com/107971872 https://vimeo.com/107971872 That said, I think a deep learning approach will likely do a lot better (and be a lot easier to develop, imo) Also, check out Google's Magenta project; it aims to use ML in various music / creativity projects. I personally plan on doing a project that will involve audio source separation as well as sample classification; a good trick for analyzing audio data is to convert it into images (maybe with some additional pre-transformations applied, such as passing it through an audio filter that exaggerates human-perceived properties of sound) and then just use your run-of-the-mill, bog-standard, state-of-the-art image classifiers on the resulting audio spectrogram with some well-chosen training/validation sets.

Eli_P 8y ago
That's interesting, are you going to make something like LabelImg[1]? I've been looking for something like that for audio, yet I'm not sure about treating audio as images. I've heard of this trick, but NN for audio better do work with RNN, GRU[2], maybe LSTM; and images are processed with CNN. [1] https://github.com/tzutalin/labelImg https://github.com/tzutalin/labelImg [2] https://en.wikipedia.org/wiki/Gated_recurrent_unit https://en.wikipedia.org/wiki/Gated_recurrent_unit
- pizza 8y ago
  I was gonna do something involving about 3 different neural nets: a source separator: taking one audio stream as input and producing a set of audio streams as output. a segmentation regression neural net: takes an audio stream as input and returns start and stop timestamps of individual samples as output, or alternativey, just trimmed copies of the audio stream sample classifier: takes an audio stream and then returns “kick drum”, “snare drum”, “voice”, “guitar”, etc then the pipeline would be like source separator => segmenter => sample classifier Hopefully with this I would be able to decompose music into constituent parts, useful for remixing and other kinds of musique concrete I expect that the results with a deep pretrained generic image model + some tweaking with more niche training examples will be satisfactory, but if not it would be a good excuse to experiment with more traditionally seuqnece-oriented network architectures
ambicapter 8y ago
Do you have to convert it into an image? What is it about classifiers that require image input? I've always found it very cool that audio compression and image compression end up using similar frequency-space techniques sometimes.
- pizza 8y ago
  You don't have to. It's just that there are a lot of good image classification network architectures that exist already so you can more or less grab a good network off-the-shelf, then give it more targeted training examples so that it performs well in your specific use-case. I guess it comes down to the fact that a lot of these signal processing techniques involve convolution.