4 ms·

can we eventually apply style transfer to speech? I'd like to hear all audio narration in the voice of David Attenborough.

georgiev 10y ago
Or Stephen Fry
- avmich 10y ago
  Or Bill Kurtis? :)
visarga 10y ago
Maybe not yet, but I am sure there will be speaker cloning extension soon, like photo and painting style transfer. Edit: It would seem they only need 25 hours of audio to train the model, so it would probably be easy to find or make alternate datasets. > We train Tacotron on an internal North American English dataset, which contains about 24.6 hours of speech data spoken by a professional female speaker
crooked-v 10y ago
Comedy option: Gilbert Gottfried.
slinger 10y ago
I was wondering the same thing! I think, in the future this might be a problem for national security
abdias 10y ago
You mean, like they did pre-2000 [1]? Don't know how good this would sound today, but nevertheless interesting. [1] http://www.washingtonpost.com/wp-srv/national/dotmil/arkin020199.htm http://www.washingtonpost.com/wp-srv/national/dotmil/arkin02...
starmole 10y ago
Adobe VoCo does something like that. Demo: https://www.youtube.com/watch?v=I3l4XLZ59iw https://www.youtube.com/watch?v=I3l4XLZ59iw
- NKCSS 10y ago
  That's pretty cool; any info on a release date?
- mbrookes 10y ago
  It was pointed out the last time I posted this that the demo is "fake" at least in as much as the inserted words aren't being synthesized but rather copy-pasted with existing intonation intact.
dharma1 10y ago
It's not really style transfer, but for a new speaker model, you just need to train each speaker with a dataset of 25 hours audio with time matched accurate transcriptions. In the case of David Attenborough, I'm pretty sure that amount of data is available with subtitled BBC content.
- dnautics 10y ago
  Oh I doubt that will be necessary. I bet you could find a useful "feature vector" by repeating the process on about 10-20 new voices - then extract a set of words (let's pessimistically say 100) that activate these features the most - then have a person retrain with 5 examples of each of these 100 words (an hour, at the most). I honestly wouldn't be surprised if you could find a really feature rich sentence (say 5 words long) that you could use to crack pretty much all voice-activated biometric password systems.
  - slinger 10y ago
    Is voice-activated password a thing? I mean, I only saw that in sci-fi movies and never in real life.
    - dharma1 10y ago
      Yes, voice biometrics are used in telephone banking for instance
  - dharma1 10y ago
    yep, I'm sure there would be a lot of similarity between models that could be used to reduce training time
gwern 10y ago
My understanding is that while a number of people have tried, we currently have no particularly viable approaches for 'style transfer' analogues on sequence data, including text and sound/speech. Samples from past work are pretty bad. But what you could possibly do is roundtrip audio to text and then text (with conditioning like speaker metadata) to speech, both of which are doable. But directly transforming, not yet.

georgiev 10y ago

Or Stephen Fry

avmich 10y ago
Or Bill Kurtis? :)

visarga 10y ago

Maybe not yet, but I am sure there will be speaker cloning extension soon, like photo and painting style transfer. Edit: It would seem they only need 25 hours of audio to train the model, so it would probably be easy to find or make alternate datasets. > We train Tacotron on an internal North American English dataset, which contains about 24.6 hours of speech data spoken by a professional female speaker

crooked-v 10y ago

Comedy option: Gilbert Gottfried.

slinger 10y ago

I was wondering the same thing! I think, in the future this might be a problem for national security

abdias 10y ago

You mean, like they did pre-2000 [1]? Don't know how good this would sound today, but nevertheless interesting. [1] http://www.washingtonpost.com/wp-srv/national/dotmil/arkin020199.htm http://www.washingtonpost.com/wp-srv/national/dotmil/arkin02...

starmole 10y ago

Adobe VoCo does something like that. Demo: https://www.youtube.com/watch?v=I3l4XLZ59iw https://www.youtube.com/watch?v=I3l4XLZ59iw

NKCSS 10y ago
That's pretty cool; any info on a release date?
mbrookes 10y ago
It was pointed out the last time I posted this that the demo is "fake" at least in as much as the inserted words aren't being synthesized but rather copy-pasted with existing intonation intact.

dharma1 10y ago

It's not really style transfer, but for a new speaker model, you just need to train each speaker with a dataset of 25 hours audio with time matched accurate transcriptions. In the case of David Attenborough, I'm pretty sure that amount of data is available with subtitled BBC content.

dnautics 10y ago
Oh I doubt that will be necessary. I bet you could find a useful "feature vector" by repeating the process on about 10-20 new voices - then extract a set of words (let's pessimistically say 100) that activate these features the most - then have a person retrain with 5 examples of each of these 100 words (an hour, at the most). I honestly wouldn't be surprised if you could find a really feature rich sentence (say 5 words long) that you could use to crack pretty much all voice-activated biometric password systems.
- slinger 10y ago
  Is voice-activated password a thing? I mean, I only saw that in sci-fi movies and never in real life.
  - dharma1 10y ago
    Yes, voice biometrics are used in telephone banking for instance
- dharma1 10y ago
  yep, I'm sure there would be a lot of similarity between models that could be used to reduce training time

gwern 10y ago

My understanding is that while a number of people have tried, we currently have no particularly viable approaches for 'style transfer' analogues on sequence data, including text and sound/speech. Samples from past work are pretty bad. But what you could possibly do is roundtrip audio to text and then text (with conditioning like speaker metadata) to speech, both of which are doable. But directly transforming, not yet.