4 ms·

I keep wondering why no-one has trained a CNN to turn low quality audio into crisp NPR sounding audio (say). Surely it’s even fairly easy to create test data fo

I keep wondering why no-one has trained a CNN to turn low quality audio into crisp NPR sounding audio (say). Surely it’s even fairly easy to create test data for such things?

notagoodidea 6y ago
I think the first step would be to be able to convert a lossy format as MP3 or ogg to a lossless format as FLAC or Wav. Being able to retrieve close in off lost data from lossy compression would be impressive.
- andrewzah 6y ago
  MP3s generally have a shelf around 16-18khz. The rest of the audible spectrum data is impossible to retrieve if it doesn't exist. This is why transcoding from a lossy->lossless format is a bad idea.
  - notagoodidea 6y ago
    Yes, I know and that's what would be more impressive for me to see a ML/Algo that could recreate close enough data distribution from lossy to lossless. Not exact reconstruction but close enough in the possible area (so < 16-18kHz). It may or may not be possible but it is more akin to take the inverse of the model of degradation of MP3 (it could be totally impossible).
    - Sohcahtoa82 6y ago
      Sounds like you're describing DLSS [0], but for audio. [0] https://en.wikipedia.org/wiki/Deep_learning_super_sampling https://en.wikipedia.org/wiki/Deep_learning_super_sampling
andrewzah 6y ago
Not everything is magically fixable. That would involve creating elements that didn't exist at recording time or were removed by an encoder.
- andy_ppp 6y ago
  With enough training data I’m sure it can be done... I’ve seen CNNs that fill in 3D scenes and animate them from two images. I would guess this was a simpler problem?
  - andrewzah 6y ago
    Well, one thing to note is that humans spend more time processing audio than video. Bad audio is immediately noticeable and aggravating, compared to spotty video with clear audio. I -guess- CNNs can look at e.g. reduced frequency range recordings (like phone calls), and attempt to reconstruct them. However this seems like an arduous mountain to climb, as people's voices are unique. So are their environments and signal chains. I really doubt that something that generalized would work very well at reconstructing a specific person's voice and recording. This also gets into the problem that it would be constructing a new reality, not recreating it.
    - andy_ppp 6y ago
      Yes, you're probably right - I hadn't thought that the voice + noise is going to be one thing, extracting just the noise will be difficult, unique and maybe not trainable at all. I have thought about the constructing a new reality thing - I wouldn't be surprised if models ended up being trained to misspeak words which could get confusing...

I keep wondering why no-one has trained a CNN to turn low quality audio into crisp NPR sounding audio (say). Surely it’s even fairly easy to create test data for such things?

notagoodidea 6y ago

I think the first step would be to be able to convert a lossy format as MP3 or ogg to a lossless format as FLAC or Wav. Being able to retrieve close in off lost data from lossy compression would be impressive.

andrewzah 6y ago
MP3s generally have a shelf around 16-18khz. The rest of the audible spectrum data is impossible to retrieve if it doesn't exist. This is why transcoding from a lossy->lossless format is a bad idea.
- notagoodidea 6y ago
  Yes, I know and that's what would be more impressive for me to see a ML/Algo that could recreate close enough data distribution from lossy to lossless. Not exact reconstruction but close enough in the possible area (so < 16-18kHz). It may or may not be possible but it is more akin to take the inverse of the model of degradation of MP3 (it could be totally impossible).
  - Sohcahtoa82 6y ago
    Sounds like you're describing DLSS [0], but for audio. [0] https://en.wikipedia.org/wiki/Deep_learning_super_sampling https://en.wikipedia.org/wiki/Deep_learning_super_sampling

andrewzah 6y ago

Not everything is magically fixable. That would involve creating elements that didn't exist at recording time or were removed by an encoder.

andy_ppp 6y ago
With enough training data I’m sure it can be done... I’ve seen CNNs that fill in 3D scenes and animate them from two images. I would guess this was a simpler problem?
- andrewzah 6y ago
  Well, one thing to note is that humans spend more time processing audio than video. Bad audio is immediately noticeable and aggravating, compared to spotty video with clear audio. I -guess- CNNs can look at e.g. reduced frequency range recordings (like phone calls), and attempt to reconstruct them. However this seems like an arduous mountain to climb, as people's voices are unique. So are their environments and signal chains. I really doubt that something that generalized would work very well at reconstructing a specific person's voice and recording. This also gets into the problem that it would be constructing a new reality, not recreating it.
  - andy_ppp 6y ago
    Yes, you're probably right - I hadn't thought that the voice + noise is going to be one thing, extracting just the noise will be difficult, unique and maybe not trainable at all. I have thought about the constructing a new reality thing - I wouldn't be surprised if models ended up being trained to misspeak words which could get confusing...