5 ms·

There is a related issue that I have some personal interest in: removing specific audio from video. Background: There are streamers on Twitch who will play co

There is a related issue that I have some personal interest in: removing specific audio from video. Background: There are streamers on Twitch who will play copyrighted music as part of their streams. Twitch allows them to do this, but then Twitch will scan these videos for copyrighted music against some music fingerprint database and mute that section of video entirely (the part of the video that played that audio). The other parts are unaffected unless they played some other copyrighted audio. YouTube does something in terms of recognizing copyrighted music, but will demonetize the entire video as a result. Needless to say, demonetization does hurt the streamer. Alternatives: I don't want to get into whether or not Twitch and YouTube are right in doing the copyrighted audio matching and the subsequent actions they take. Some streamers who've been affected by this have started playing royalty-free/copyright-free music or music from lesser known artists that are less likely to be in these music fingerprint databases. My question: Is it possible to just subtract the audio of a copyrighted track from a video after it has been detected to having being played in a video?

Hydraulix989 7y ago
It's possible, but the quality isn't that great even with state of the art machine learning: https://towardsdatascience.com/audio-ai-isolating-vocals-from-stereo-music-using-convolutional-neural-networks-210532383785 https://towardsdatascience.com/audio-ai-isolating-vocals-fro...
- canada_dry 7y ago
  The only tool in the audio toolbox is essentially the fourier transform. It would be a game changer if someone were to come up with a novel method of decomposing audio into discrete components (e.g. people speaking, specific instruments, background noise). Likely this would require completely new hardware to capture different audio attributes in addition to simply capturing a stream of vibrations from a microphone.
  - tareqak 7y ago
    I think something imperfect would be welcome given the current set of consequences: just removing the Fourier Transform of a song given a start time and and end time should get rid off enough to keep copyright holders and content creators somewhat happy.
  - kyzyl 7y ago
    The tools to do this exist. It's usually called 'blind source separation', as in "What are the N distinct audio signals which sum up to best explain a given compound signal, without knowing the possible source signals ahead of time." Usually it's done with some sort of matrix factorization, Principal Component Analysis, and/or Independent Component Analysis. It's also used for non-audio signals, like pulling the discrete firings out of noisy EEG signals. It's definitely not a foolproof solution but in a lot of applications it can get you going, at least.
    - salty_biscuits 7y ago
      By the problem setup it isn't blind source. It is sound = song plus other. A mixture model with 2 components Edit. If you know the song it should be something simple like do cross correlation of audio with known song. Find peak. Solve for the gain and subtract away scaled and shifted song from original track. Will be rubbish if gain and timing have errors. Might need to do it in little chunks and interpolate the gain and shifts. Edit 2. More generally, you might want to worry about the song having passed through some unknown transfer function (i.e. it is being played and recorded through shitty equipment). Then you have an interesting inverse problem. If everything is linear it will involve a regularized deconvolution. Will be tricky then.
      - Hydraulix989 7y ago
        It still is reduce-able to the more general blind source problem, right? We can conveniently "forget" that we know what the sources are so now we are blindfolded and can still use the same techniques to solve it.
taneq 7y ago
Is AI the right tool for this? It seems more akin to traditional noise cancellation, where the "noise" is the track you want to remove.
tinus_hn 7y ago
If the track is decompressed by the game, played and then recompressed by the streamer, you can’t just subtract it because that will leave a garbled mix of all the things thrown away by the lossy compression. It might be possible to determine what parts of the reencoded stream are the the copyrighted track though and drop these. If you were able to do this perfectly you would still lose quality but not that much.
teilo 7y ago
This is very difficult at present. Most methods of doing something like this rely upon spectral analysis of the target you wish to remove, and then use FFT decomp/recomp to subtract the result. The results are poor. Someone mentioned the software that can isolate instruments from one another. But the problem is this usually relies upon the instruments or vocals having a different position in a stereo signal, and different frequency ranges, to perform that extraction. This is much easer to do in such cases, especially since the material you are working with is generally clean source material. Here you have a mixed signal, that may or may not contain stereo data, but certainly contains a lot of background noise, as well as other game audio you wish to retain. What is needed is a far more intelligent method of identifying the spectral signature of a known source by matching the target with the source in the time domain, and then using that identified signal in the target itself to cancel itself out. In practice this would mean comparing the amplitude modulation of each spectral band in the source to the target to identify the spectral components that need to be removed, and then using the corresponding "matched" bands, generate a signal, using the source as a guide, from the target audio itself that represents only that portion of the spectral signature that is being matched to the source. This newly generated signal can then be used to remove that audio from the target with 180º phase cancellation. Keep in mind that wherever the spectral bands overlap with other audio content you wish to retain, there will be a lot of artifacting and signal loss or phasing. A second pass would need to use the equivalent of inpainting to reconstruct those missing components. Overall, it's a very hard problem.

Hydraulix989 7y ago

It's possible, but the quality isn't that great even with state of the art machine learning: https://towardsdatascience.com/audio-ai-isolating-vocals-from-stereo-music-using-convolutional-neural-networks-210532383785 https://towardsdatascience.com/audio-ai-isolating-vocals-fro...

canada_dry 7y ago
The only tool in the audio toolbox is essentially the fourier transform. It would be a game changer if someone were to come up with a novel method of decomposing audio into discrete components (e.g. people speaking, specific instruments, background noise). Likely this would require completely new hardware to capture different audio attributes in addition to simply capturing a stream of vibrations from a microphone.
- tareqak 7y ago
  I think something imperfect would be welcome given the current set of consequences: just removing the Fourier Transform of a song given a start time and and end time should get rid off enough to keep copyright holders and content creators somewhat happy.
- kyzyl 7y ago
  The tools to do this exist. It's usually called 'blind source separation', as in "What are the N distinct audio signals which sum up to best explain a given compound signal, without knowing the possible source signals ahead of time." Usually it's done with some sort of matrix factorization, Principal Component Analysis, and/or Independent Component Analysis. It's also used for non-audio signals, like pulling the discrete firings out of noisy EEG signals. It's definitely not a foolproof solution but in a lot of applications it can get you going, at least.
  - salty_biscuits 7y ago
    By the problem setup it isn't blind source. It is sound = song plus other. A mixture model with 2 components Edit. If you know the song it should be something simple like do cross correlation of audio with known song. Find peak. Solve for the gain and subtract away scaled and shifted song from original track. Will be rubbish if gain and timing have errors. Might need to do it in little chunks and interpolate the gain and shifts. Edit 2. More generally, you might want to worry about the song having passed through some unknown transfer function (i.e. it is being played and recorded through shitty equipment). Then you have an interesting inverse problem. If everything is linear it will involve a regularized deconvolution. Will be tricky then.
    - Hydraulix989 7y ago
      It still is reduce-able to the more general blind source problem, right? We can conveniently "forget" that we know what the sources are so now we are blindfolded and can still use the same techniques to solve it.

taneq 7y ago

Is AI the right tool for this? It seems more akin to traditional noise cancellation, where the "noise" is the track you want to remove.

tinus_hn 7y ago

If the track is decompressed by the game, played and then recompressed by the streamer, you can’t just subtract it because that will leave a garbled mix of all the things thrown away by the lossy compression. It might be possible to determine what parts of the reencoded stream are the the copyrighted track though and drop these. If you were able to do this perfectly you would still lose quality but not that much.

teilo 7y ago

This is very difficult at present. Most methods of doing something like this rely upon spectral analysis of the target you wish to remove, and then use FFT decomp/recomp to subtract the result. The results are poor. Someone mentioned the software that can isolate instruments from one another. But the problem is this usually relies upon the instruments or vocals having a different position in a stereo signal, and different frequency ranges, to perform that extraction. This is much easer to do in such cases, especially since the material you are working with is generally clean source material. Here you have a mixed signal, that may or may not contain stereo data, but certainly contains a lot of background noise, as well as other game audio you wish to retain. What is needed is a far more intelligent method of identifying the spectral signature of a known source by matching the target with the source in the time domain, and then using that identified signal in the target itself to cancel itself out. In practice this would mean comparing the amplitude modulation of each spectral band in the source to the target to identify the spectral components that need to be removed, and then using the corresponding "matched" bands, generate a signal, using the source as a guide, from the target audio itself that represents only that portion of the spectral signature that is being matched to the source. This newly generated signal can then be used to remove that audio from the target with 180º phase cancellation. Keep in mind that wherever the spectral bands overlap with other audio content you wish to retain, there will be a lot of artifacting and signal loss or phasing. A second pass would need to use the equivalent of inpainting to reconstruct those missing components. Overall, it's a very hard problem.