4 ms·

While it's a great technology, the result sounds somewhat robotic. On the original recording the voice sounds soft, but after separation it sounds like it is sy

While it's a great technology, the result sounds somewhat robotic. On the original recording the voice sounds soft, but after separation it sounds like it is synthesized or passed through a vocoder, something is missing. The voice contains pieces of strumming sound. Guitar also sounds "blurred", as if someone cut an object from the picture and blurred the cut to make it less visible. Clap sound is distorted, on the original recording it sounds the same, but after separation it sounds different every time, as if it was filtered or compressed with low bitrate. It is amazing how the ear manages to distinguish all the sounds without distortion.

MattRix 7y ago
It's easy to pick up on lots of little things, but this is still extremely impressive. It's also more than good enough for people who want to practice singing/playing over the song by themselves. I think the next step would be to train a network that can un-robotify songs and then run it on this.
uryga 7y ago
> On the original recording the voice sounds soft, but after separation it sounds like it is synthesized or passed through a vocoder funny, the original recording seemed kind of robotic to me! maybe not robotic, but like it's been filtered somehow. but that might just be my not-so-great headphones
- mwcampbell 7y ago
  The original lead vocal is definitely processed. You can hear that processing clearly if you apply the classic vocal removal (really, center removal) effect. (My favorite implementation of that is the Center Cut DSP for foobar2000.)
nickjj 7y ago
That roboticness is because there's overlap in frequencies between the voice and the instruments. I have no idea how this tool splits them up at the implementation level but I imagine it tries to split it up based on frequencies and when it lifts out the voice, it's cutting out a ton of frequencies that would normally be in your voice so now it sounds very unnatural, blocky and metallic. With studio quality headphones I can notice a massive difference for the worse between the original and separated vocal track. It reminds me of when I turned on a noise gate too high when recording audio for my courses. That noise gate clamps down on certain frequencies to help eliminate room noise but it also dampens or removes natural frequencies that occur on the low end of an average male voice. It gives that same very jagged sounding audio waveform.
- codedokode 7y ago
  I don't argue that it's a great technology, but some of the neighbour commenters wrote things like "perfect", and to me it doesn't sound like "perfect" yet. For example, in the beginning, if you listen to the phrase "from the office ...", in original recording it sounds smooth, like a single phrase (and the voice is warm and pleasant), but in the separated track it sounds like it is synthesized from pieces that are not properly connected, like vocaloid songs. It sounds little harsher. Transitions sound unnatural. And the phrase "heya Tom" in the separated track is split into "heya" and "Tom" with some unnatural sound between (or maybe in the beginning of "Tom"). It is like transitions you can hear in vocaloid tracks. Or the kind of artifacts you get if you over-compress an MP3 file. And "it's good to see you" part also sounds robotic. Maybe it's losing some of harmonics, but in a different way for different syllables and they don't sound like a single phrase anymore.
  - sjwright 7y ago
    It’s perfect relative to the most wildly optimistic expectations anyone could have reasonably held beforehand. The vocals don’t have any significant residual artefacts from drum hits or any residual bleed-through of instruments playing the same notes. It’s magical.
jacquesm 7y ago
> While it's a great technology, the result sounds somewhat robotic. That's like complaining about how bad the pig plays the violin. This is absolutely incredible. The complexity level for this problem is right off the scale and the software does a passable job of it. Given some time and more training data and a few more people working on it this has serious potential.
- mikorym 7y ago
  I think you meant how well the pig paints. [1] [2] [1] https://pigcasso.org/wp-content/uploads/2018/12/7.jpg https://pigcasso.org/wp-content/uploads/2018/12/7.jpg [2] Also, I am not affiliated with anything in particular that has been mentioned, or with the pig.
  - jacquesm 7y ago
    I think the original was in relation to pigs dancing.
zzzeek 7y ago
this is like someone just flew you to the surface of Mars in an hour and your only comment is that the ride was bumpy. The demo above is mind blowing. I basically want to run this over every Steve Gadd recording I have.
jononor 7y ago
People are hard at work on denoising of audio using neural networks also. I would expect that if one trained denoisers for each type of source separated here, and passed the separated audio through them, things would get even better.
- rrss 7y ago
  spleeter is using neural networks.
  - jononor 7y ago
    Yes, for source separation. Denoising is generally a separate task. A denoising network would take in a noisy signal for a single source and outputs a cleaned up one. It would be trained on the specific source type, for example vocals.
    - rrss 7y ago
      Gotcha, thanks.
rafa1981 7y ago
Maybe the source was compressed audio instead of flac/wav? Edit: the source is an mp3, which removes audio frequencies based on perception/masking with other frequencies. It's perfectly normal that it is showing artifacts. A better source is needed.
- mwcampbell 7y ago
  I encoded the source in MP3 for your convenience. But the source that I fed into Spleeter was FLAC. Edit: I uploaded the original FLAC to my web server last night, before I decided that MP3 would be more convenient: https://mwcampbell.us/tmp/spleeter-demo/jonathan-coulton-re-your-brains/original.flac https://mwcampbell.us/tmp/spleeter-demo/jonathan-coulton-re-...

MattRix 7y ago

It's easy to pick up on lots of little things, but this is still extremely impressive. It's also more than good enough for people who want to practice singing/playing over the song by themselves. I think the next step would be to train a network that can un-robotify songs and then run it on this.

uryga 7y ago

> On the original recording the voice sounds soft, but after separation it sounds like it is synthesized or passed through a vocoder funny, the original recording seemed kind of robotic to me! maybe not robotic, but like it's been filtered somehow. but that might just be my not-so-great headphones

mwcampbell 7y ago
The original lead vocal is definitely processed. You can hear that processing clearly if you apply the classic vocal removal (really, center removal) effect. (My favorite implementation of that is the Center Cut DSP for foobar2000.)

nickjj 7y ago

That roboticness is because there's overlap in frequencies between the voice and the instruments. I have no idea how this tool splits them up at the implementation level but I imagine it tries to split it up based on frequencies and when it lifts out the voice, it's cutting out a ton of frequencies that would normally be in your voice so now it sounds very unnatural, blocky and metallic. With studio quality headphones I can notice a massive difference for the worse between the original and separated vocal track. It reminds me of when I turned on a noise gate too high when recording audio for my courses. That noise gate clamps down on certain frequencies to help eliminate room noise but it also dampens or removes natural frequencies that occur on the low end of an average male voice. It gives that same very jagged sounding audio waveform.

codedokode 7y ago
I don't argue that it's a great technology, but some of the neighbour commenters wrote things like "perfect", and to me it doesn't sound like "perfect" yet. For example, in the beginning, if you listen to the phrase "from the office ...", in original recording it sounds smooth, like a single phrase (and the voice is warm and pleasant), but in the separated track it sounds like it is synthesized from pieces that are not properly connected, like vocaloid songs. It sounds little harsher. Transitions sound unnatural. And the phrase "heya Tom" in the separated track is split into "heya" and "Tom" with some unnatural sound between (or maybe in the beginning of "Tom"). It is like transitions you can hear in vocaloid tracks. Or the kind of artifacts you get if you over-compress an MP3 file. And "it's good to see you" part also sounds robotic. Maybe it's losing some of harmonics, but in a different way for different syllables and they don't sound like a single phrase anymore.
- sjwright 7y ago
  It’s perfect relative to the most wildly optimistic expectations anyone could have reasonably held beforehand. The vocals don’t have any significant residual artefacts from drum hits or any residual bleed-through of instruments playing the same notes. It’s magical.

jacquesm 7y ago

> While it's a great technology, the result sounds somewhat robotic. That's like complaining about how bad the pig plays the violin. This is absolutely incredible. The complexity level for this problem is right off the scale and the software does a passable job of it. Given some time and more training data and a few more people working on it this has serious potential.

mikorym 7y ago
I think you meant how well the pig paints. [1] [2] [1] https://pigcasso.org/wp-content/uploads/2018/12/7.jpg https://pigcasso.org/wp-content/uploads/2018/12/7.jpg [2] Also, I am not affiliated with anything in particular that has been mentioned, or with the pig.
- jacquesm 7y ago
  I think the original was in relation to pigs dancing.

zzzeek 7y ago

this is like someone just flew you to the surface of Mars in an hour and your only comment is that the ride was bumpy. The demo above is mind blowing. I basically want to run this over every Steve Gadd recording I have.

jononor 7y ago

People are hard at work on denoising of audio using neural networks also. I would expect that if one trained denoisers for each type of source separated here, and passed the separated audio through them, things would get even better.

rrss 7y ago
spleeter is using neural networks.
- jononor 7y ago
  Yes, for source separation. Denoising is generally a separate task. A denoising network would take in a noisy signal for a single source and outputs a cleaned up one. It would be trained on the specific source type, for example vocals.
  - rrss 7y ago
    Gotcha, thanks.

rafa1981 7y ago

Maybe the source was compressed audio instead of flac/wav? Edit: the source is an mp3, which removes audio frequencies based on perception/masking with other frequencies. It's perfectly normal that it is showing artifacts. A better source is needed.

mwcampbell 7y ago
I encoded the source in MP3 for your convenience. But the source that I fed into Spleeter was FLAC. Edit: I uploaded the original FLAC to my web server last night, before I decided that MP3 would be more convenient: https://mwcampbell.us/tmp/spleeter-demo/jonathan-coulton-re-your-brains/original.flac https://mwcampbell.us/tmp/spleeter-demo/jonathan-coulton-re-...