8 ms·

Open Source Tools and Data for Music Source Separation

277 points by homarp 6y ago

lukax 6y ago
Deezer also open-sourced their source separation library Spleeter [1][2] last year (submitted to ISMIR2019 [3]). Deezer does not offer an official hosted service but there are unofficial services like Acapella Extractor [4], melody ml [5] and moises [6]. [1] https://news.ycombinator.com/item?id=21431071 https://news.ycombinator.com/item?id=21431071 [2] https://news.ycombinator.com/item?id=23228539 https://news.ycombinator.com/item?id=23228539 [3] http://archives.ismir.net/ismir2019/latebreaking/000036.pdf http://archives.ismir.net/ismir2019/latebreaking/000036.pdf [4] https://www.acapella-extractor.com/ https://www.acapella-extractor.com/ [5] https://melody.ml/ https://melody.ml/ [6] https://moises.ai/ https://moises.ai/
- zzzeek 6y ago
  I was playing with spleeter a bit earlier in the year for drum practice and it was great. You can do drum covers (or whatever instrument) without the other (typically famous and amazing ) musician stepping all over your brilliant amateur playing !
- pseeth 6y ago
  Hi there! One of the authors here (Prem). Thanks for reading, and for mentioning Spleeter! Spleeter is awesome. The idea behind this book is to give people the tools and code to train their own versions of Spleeter. And, great links! We'll add them to our Appendix.
- The_rationalist 6y ago
  Spleeter is no longer the SOTA https://paperswithcode.com/sota/music-source-separation-on-musdb18 https://paperswithcode.com/sota/music-source-separation-on-m...
  - tomduncalf 6y ago
    Nice, I just tried Demucs on my go to track (Fleetwood Mac - Dreams, mainly because it happened to be on my computer and I thought I could hear it multiple times without going crazy, but also it does seem to be a reasonably good test as there are some effects which seem to make separating the voice non-trivial etc) and the end result was very impressive. Definitely much cleaner sounding (sharper and fewer FFT artefacts) than Spleeter, especially on the drums and vocal. I also tried it on a dance track (BT - Mercury and Solace) and the results were much less impressive. I guess this maybe reflects what it was trained on?
- JeffreyCA 6y ago
  There's also Spleeter Web [1], a self-hostable version of a Spleeter web app. A fun little project I've been working on! I'm thinking of supporting additional models such as Demucs [2], which seems to perform better than Spleeter. [1] https://github.com/JeffreyCA/spleeter-web https://github.com/JeffreyCA/spleeter-web [2] https://github.com/facebookresearch/demucs https://github.com/facebookresearch/demucs
  - djeiasbsbo 6y ago
    That is absolutely awesome and exactly what I've been looking for. Thank you for working on this and showing it here!
agbell 6y ago
Wow, great guide ... I just skimmed it. Izotope RX,which is mentioned, in the appendix, is amazing and I use it to separate noise from vocals for my podcast. The dereverb that it can do is pretty amazing.
- jsal 6y ago
  One of the authors here (justin), thanks for the feedback! The goal of this tutorial is to show how you can use open-source tools to create your own RX :) For example, you can train it to remix/separate a specific set of instruments you care about. Cheers!
matheusmoreira 6y ago
I always wondered why music files don't have separate channels for each instrument or voice. All the sounds are mixed into one or two channels. Why? Wouldn't it be better to do this as late as possible? Watching waveforms that actually match what I'm hearing is so pleasant... https://youtu.be/eclMFa0mD1c https://youtu.be/eclMFa0mD1c
- TeMPOraL 6y ago
  Same here. I'm guessing there are two reasons. One, legacy: that in the early days of digital sound, composing tracks in real-time on end-user machines was too computationally expensive. Two, IP: having separate channels for every instrument or voice makes it easier to play with and remix the music, which is something IP holders don't want.
- hunter2_ 6y ago
  Some music is released this way: https://en.m.wikipedia.org/wiki/List_of_musical_works_released_in_a_stem_format https://en.m.wikipedia.org/wiki/List_of_musical_works_releas... Obviously this is nowhere near a comprehensive list. That said, it's definitely rare.
- anamexis 6y ago
  The simple answer is that humans have two channels of audio in.
- jacquesm 6y ago
  They exist, and if you know where to look you will find many of them, almost always not legal. The storage requirements are vast and you will need some pretty specialized hardware to do playback. The biggest reason why stuff is downmixed and flattened is because of our historical distribution media: radio, tape, records, in theory you could distribute the masters but most people would have no use for them.
  - kzrdude 6y ago
    Some of the youtubers that talk about and analyze music seem to be able to access multi-track recordings of various well-known songs, I was always wondering where they got them from.
mxmilkb 6y ago
Related; https://github.com/linuxmao-org/ISSE https://github.com/linuxmao-org/ISSE https://github.com/etzinis/sudo_rm_rf https://github.com/etzinis/sudo_rm_rf - SUccessive DOwnsampling and Resampling of Multi-Resolution Features which enables a more efficient way of separating sources from mixtures https://github.com/adobe/Deep-Audio-Prior https://github.com/adobe/Deep-Audio-Prior - Audio Source Separation Without Any Training Data
pabs3 6y ago
This reminds me of this open source project (and its predecessor manyears and open hardware projects 8/16soundsusb). https://github.com/introlab/odas https://github.com/introlab/odas https://github.com/introlab/manyears https://github.com/introlab/manyears https://github.com/introlab/16SoundsUSB https://github.com/introlab/16SoundsUSB Website of the team behind these: https://introlab.3it.usherbrooke.ca/ https://introlab.3it.usherbrooke.ca/

277 points by homarp 6y ago

Deezer also open-sourced their source separation library Spleeter [1][2] last year (submitted to ISMIR2019 [3]). Deezer does not offer an official hosted service but there are unofficial services like Acapella Extractor [4], melody ml [5] and moises [6]. [1] https://news.ycombinator.com/item?id=21431071 https://news.ycombinator.com/item?id=21431071 [2] https://news.ycombinator.com/item?id=23228539 https://news.ycombinator.com/item?id=23228539 [3] http://archives.ismir.net/ismir2019/latebreaking/000036.pdf http://archives.ismir.net/ismir2019/latebreaking/000036.pdf [4] https://www.acapella-extractor.com/ https://www.acapella-extractor.com/ [5] https://melody.ml/ https://melody.ml/ [6] https://moises.ai/ https://moises.ai/

zzzeek 6y ago
I was playing with spleeter a bit earlier in the year for drum practice and it was great. You can do drum covers (or whatever instrument) without the other (typically famous and amazing ) musician stepping all over your brilliant amateur playing !
pseeth 6y ago
Hi there! One of the authors here (Prem). Thanks for reading, and for mentioning Spleeter! Spleeter is awesome. The idea behind this book is to give people the tools and code to train their own versions of Spleeter. And, great links! We'll add them to our Appendix.
The_rationalist 6y ago
Spleeter is no longer the SOTA https://paperswithcode.com/sota/music-source-separation-on-musdb18 https://paperswithcode.com/sota/music-source-separation-on-m...
- tomduncalf 6y ago
  Nice, I just tried Demucs on my go to track (Fleetwood Mac - Dreams, mainly because it happened to be on my computer and I thought I could hear it multiple times without going crazy, but also it does seem to be a reasonably good test as there are some effects which seem to make separating the voice non-trivial etc) and the end result was very impressive. Definitely much cleaner sounding (sharper and fewer FFT artefacts) than Spleeter, especially on the drums and vocal. I also tried it on a dance track (BT - Mercury and Solace) and the results were much less impressive. I guess this maybe reflects what it was trained on?
JeffreyCA 6y ago
There's also Spleeter Web [1], a self-hostable version of a Spleeter web app. A fun little project I've been working on! I'm thinking of supporting additional models such as Demucs [2], which seems to perform better than Spleeter. [1] https://github.com/JeffreyCA/spleeter-web https://github.com/JeffreyCA/spleeter-web [2] https://github.com/facebookresearch/demucs https://github.com/facebookresearch/demucs
- djeiasbsbo 6y ago
  That is absolutely awesome and exactly what I've been looking for. Thank you for working on this and showing it here!

agbell 6y ago

Wow, great guide ... I just skimmed it. Izotope RX,which is mentioned, in the appendix, is amazing and I use it to separate noise from vocals for my podcast. The dereverb that it can do is pretty amazing.

jsal 6y ago
One of the authors here (justin), thanks for the feedback! The goal of this tutorial is to show how you can use open-source tools to create your own RX :) For example, you can train it to remix/separate a specific set of instruments you care about. Cheers!

matheusmoreira 6y ago

I always wondered why music files don't have separate channels for each instrument or voice. All the sounds are mixed into one or two channels. Why? Wouldn't it be better to do this as late as possible? Watching waveforms that actually match what I'm hearing is so pleasant... https://youtu.be/eclMFa0mD1c https://youtu.be/eclMFa0mD1c

TeMPOraL 6y ago
Same here. I'm guessing there are two reasons. One, legacy: that in the early days of digital sound, composing tracks in real-time on end-user machines was too computationally expensive. Two, IP: having separate channels for every instrument or voice makes it easier to play with and remix the music, which is something IP holders don't want.
hunter2_ 6y ago
Some music is released this way: https://en.m.wikipedia.org/wiki/List_of_musical_works_released_in_a_stem_format https://en.m.wikipedia.org/wiki/List_of_musical_works_releas... Obviously this is nowhere near a comprehensive list. That said, it's definitely rare.
anamexis 6y ago
The simple answer is that humans have two channels of audio in.
jacquesm 6y ago
They exist, and if you know where to look you will find many of them, almost always not legal. The storage requirements are vast and you will need some pretty specialized hardware to do playback. The biggest reason why stuff is downmixed and flattened is because of our historical distribution media: radio, tape, records, in theory you could distribute the masters but most people would have no use for them.
- kzrdude 6y ago
  Some of the youtubers that talk about and analyze music seem to be able to access multi-track recordings of various well-known songs, I was always wondering where they got them from.

mxmilkb 6y ago

Related; https://github.com/linuxmao-org/ISSE https://github.com/linuxmao-org/ISSE https://github.com/etzinis/sudo_rm_rf https://github.com/etzinis/sudo_rm_rf - SUccessive DOwnsampling and Resampling of Multi-Resolution Features which enables a more efficient way of separating sources from mixtures https://github.com/adobe/Deep-Audio-Prior https://github.com/adobe/Deep-Audio-Prior - Audio Source Separation Without Any Training Data

pabs3 6y ago

This reminds me of this open source project (and its predecessor manyears and open hardware projects 8/16soundsusb). https://github.com/introlab/odas https://github.com/introlab/odas https://github.com/introlab/manyears https://github.com/introlab/manyears https://github.com/introlab/16SoundsUSB https://github.com/introlab/16SoundsUSB Website of the team behind these: https://introlab.3it.usherbrooke.ca/ https://introlab.3it.usherbrooke.ca/