7 ms·

The Rise of Synthetic Audio Deepfakes

183 points by ajay-d 6y ago

nshm 6y ago
> Deepfake technology is not sophisticated enough to mimic an entire phone call with someone. With modern voice conversion technology it is perfectly possible actually.
- seesawtron 6y ago
  indeed, the text to speech conversation works extremely fast even in browser (like Colab) once you get your model tuned right.
  - jcims 6y ago
    Kids in 2020 are going to be doing prank phone calls with GPT-3 and voice models from untraceable SIP phone numbers.
    - holoduke 6y ago
      and Facebook is currently generating billions of voice profiles based on recorded WhatsApp sessions. Soon these profiles are sold to advertising agencies.
      - smegma2 6y ago
        Is this true? On Google I only see mentions of using voice data to improve their speech transcribing.
      - personjerry 6y ago
        No, WhatsApp messages and calls are end-to-end encrypted.
        jcims 6y ago
        Sounds like a conspiracy theory but those aren’t mutually exclusive.
- crazygringo 6y ago
  Voice imitation isn't just timbre. It's prosody, rhythm, accent, word choice, and so on. By the time you've mastered all those, you're practically halfway to becoming a professional voiceover artist. Remember, trained voiceover artists have been mimicking voices for a long, long time. Their timbre isn't always perfect, but faking voices doesn't need deepfakes.
  - Enginerrrd 6y ago
    If you're looking for it sure. But I'm willing to bet existing technology is sufficient to catch an awful lot of people off guard. Hearing a familiar voice is usually quite disarming.
seesawtron 6y ago
This makes me wonder how would one go about adding an authentication key to audio? We have seen in the past encryption for text shared via email and watermarks embedded in images but I haven't come across something for audio. Happy to hear if someone has worked in this field.
- nine_k 6y ago
  > adding an authentication key to audio I'm afraid that it's not going to work. Not more than adding an authentication key to handwriting. The voice itself was an authentication key, as was the hand. But now these keys are too weak and easy to forge. I'd say we should completely stop trusting the authenticity of any recorded voice, and maybe all voice by phone. Trust the content, the style, have a set of agreed-upon keywords with your partner and/or your parents to check that the other side is indeed who it alleges to be in an extraordinary situation. Maybe not today, but tomorrow it may happen to be useful to thwart a scam directed against you, with the scammer perfectly imitating the voice of your loved one(s).
motohagiography 6y ago
Useful example is how the Joe Biden Burisma phone call that bubbled up through Russian media was fabricated. I pulled it apart with ffmpeg and there were a number of artifacts that showed editing and splicing. If you're handy with ffmpeg and python, you can assess their veracity pretty easily. Of course, if I were on a political ratf'ing team, I'd use the same tools to add those artifacts to a copy of an offending (real but off message) stream and amplify the distribution of that fake-faked version with a debunking press release handy, so YMMV. While the Biden thing wasn't a deepfake directly, (shallow fake?) we're going to see tons of actual deepfakes around the election. IMO, elections are no longer between candidates, they are a war on truth for domination of the narrative - office is the effect. A campaign that focuses on what happens once the war is over is daydreaming about the future and distracted from the present and this will lose them key battles. For this reason, I think deepfakes are going to be the biggest weapon in campaign arsenals for the near future. Interesting times.
- pantaloony 6y ago
  I’m legitimately not sure democracy will survive modern, sophisticated propaganda techniques, plus an open, international Web, plus losing the ability to more-or-less trust audio and video recordings that we’ve grown used to over the last hundredish years. Between state actors and, eventually if not already, transnational corporations waging info warfare, I kinda doubt the institution can take it. Too much info, too fast, from too many sources.
  - brutt 6y ago
    It looks like problem! :-) I have solution for this for years, but demand is not there yet.
    - ethbro 6y ago
      The solution is to make media literacy and logical fallacies a required course in democratic basic education.
      - 6y ago
yalogin 6y ago
There is already a large problem with political ads cherry picking and slicing audio and video to cheat viewers. I really worry that deep fakes will take it to another level completely. I fully expect the current administration to eagerly adopt it if available.
- deleted 6y ago
  [deleted]
- acruns 6y ago
  I just assume if a politician is speaking they are lying, at best misleading. Add in marketing, I assume it is a lie, regardless of political party.
  - praptak 6y ago
    You still lose. There are players who don't need you to believe them, as it is sufficient for them if people cannot trust each other.
- notahacker 6y ago
  People have been impersonating politicians' accents pretty well for decades, mostly for comedy value, without tripping over sentence structures and pronunciation (unless that's the joke...) Other than lowering the effort bar, I'm not sure what deepfakes add. Sure, it means someone can make Donald Trump reading copypasta[1] or Dubya rapping [2] without knowing how to do impressions or even speaking the language natively, but a competent voice actor could do a much better job of putting words into politicians' mouths, and one thing political ads don't lack is budget. 1. https://www.youtube.com/watch?v=LEzIAixNkFI https://www.youtube.com/watch?v=LEzIAixNkFI 2. https://www.youtube.com/watch?v=LEzIAixNkFI https://www.youtube.com/watch?v=LEzIAixNkFI
paul7986 6y ago
Recently a friend changed her number and told me via text. Before adding her number i asked her a question that she and only I would know like who sat next to you at the old office. Think im going to keep doing this type of verification. It may annoy friends and family, but not sure how a hacker could ever know such small details between you and another.
- Wistar 6y ago
  Very good idea. A few years ago, I had a "friend" text me asking for emergency funds. It seemed wonky but within the realm of possibility. I asked a question about a mutual friend from the past revealed it as a scam.
leptoniscool 6y ago
How long does it take to train the model?
sidthekid 6y ago
The images of spectrogram analysis between the real and fake voices seemed to be distinguishable by the human eye. Can a image model be trained to detect fake voice spectrograms based on pitch and tone choppiness?
- grogenaut 6y ago
  Would you not just then take this and feed it into the training?
- mensetmanusman 6y ago
  The issue is that if you can measure it, you can train an AI to beat the other AI detecting it. As Pilate said: ‘Quid est veritas?‘
  - deepnotderp 6y ago
    Generation is a much harder problem than discrimination though.
phjesusthatguy3 6y ago
Audio "deepfakes" have been worked on much longer than ones for video, although video deepfakes have the added issue of deep-faking synchronized audio. Today's consumers don't seem to be bothered by video deepfakes if they play to the beliefs of the audience.
inasio 6y ago
Next milestone I'm waiting for: Trump audio to Sarah Cooper video on the fly
eigenvalue 6y ago
My all-time favorite audio deepfake is Nobel prize winner Milton Friedman reading the lyrics to the 50 Cent track "PIMP". It really captures Friedman's tell-tale cadence and idiosyncratic lilt: https://www.youtube.com/watch?v=4mUYMvuNIas https://www.youtube.com/watch?v=4mUYMvuNIas
blueblisters 6y ago
There is an annual challenge for synthetic voice detection, ASVSpoof, that evaluates submissions on different types of attacks to speaker verification systems: text to speech, voice conversions and replayed attacks. The conclusion from the 2019 evaluation [1]: known synthetic deep fakes are fairly easy to detect using simple models with very low error rates (even high-fidelity techniques with Wavenet vocoders). [1]: ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection speech (https://www.isca-speech.org/archive/Interspeech_2019/pdfs/2249.pdf https://www.isca-speech.org/archive/Interspeech_2019/pdfs/22...)
kharak 6y ago
This might be paranoid. But I've established a protocol with some people in my life. Should someone with my voice ever contact them and ask for money (because emergency bla bla), nothing is to be done until a passphrase is mentioned. It's only a matter of time, until someone gets significant voice data and related contact numbers and proceeds with using those voices to train a model. Afterwards, that model will be used to real-time fake the original voice in a scamming attempt.
- grumio 6y ago
  That's good practice. Sadly these kinds of scams already happen without the effort of synthetic voices. Scammers call an older person and say "Hey Grandma, it's me your grandson. I need $500 for bail right away!" With the help of Facebook they can learn names and details to sound more convincing.

183 points by ajay-d 6y ago

nshm 6y ago

> Deepfake technology is not sophisticated enough to mimic an entire phone call with someone. With modern voice conversion technology it is perfectly possible actually.

seesawtron 6y ago
indeed, the text to speech conversation works extremely fast even in browser (like Colab) once you get your model tuned right.
- jcims 6y ago
  Kids in 2020 are going to be doing prank phone calls with GPT-3 and voice models from untraceable SIP phone numbers.
  - holoduke 6y ago
    and Facebook is currently generating billions of voice profiles based on recorded WhatsApp sessions. Soon these profiles are sold to advertising agencies.
    - smegma2 6y ago
      Is this true? On Google I only see mentions of using voice data to improve their speech transcribing.
    - personjerry 6y ago
      No, WhatsApp messages and calls are end-to-end encrypted.
      - jcims 6y ago
        Sounds like a conspiracy theory but those aren’t mutually exclusive.
crazygringo 6y ago
Voice imitation isn't just timbre. It's prosody, rhythm, accent, word choice, and so on. By the time you've mastered all those, you're practically halfway to becoming a professional voiceover artist. Remember, trained voiceover artists have been mimicking voices for a long, long time. Their timbre isn't always perfect, but faking voices doesn't need deepfakes.
- Enginerrrd 6y ago
  If you're looking for it sure. But I'm willing to bet existing technology is sufficient to catch an awful lot of people off guard. Hearing a familiar voice is usually quite disarming.

seesawtron 6y ago

This makes me wonder how would one go about adding an authentication key to audio? We have seen in the past encryption for text shared via email and watermarks embedded in images but I haven't come across something for audio. Happy to hear if someone has worked in this field.

nine_k 6y ago
> adding an authentication key to audio I'm afraid that it's not going to work. Not more than adding an authentication key to handwriting. The voice itself was an authentication key, as was the hand. But now these keys are too weak and easy to forge. I'd say we should completely stop trusting the authenticity of any recorded voice, and maybe all voice by phone. Trust the content, the style, have a set of agreed-upon keywords with your partner and/or your parents to check that the other side is indeed who it alleges to be in an extraordinary situation. Maybe not today, but tomorrow it may happen to be useful to thwart a scam directed against you, with the scammer perfectly imitating the voice of your loved one(s).

motohagiography 6y ago

Useful example is how the Joe Biden Burisma phone call that bubbled up through Russian media was fabricated. I pulled it apart with ffmpeg and there were a number of artifacts that showed editing and splicing. If you're handy with ffmpeg and python, you can assess their veracity pretty easily. Of course, if I were on a political ratf'ing team, I'd use the same tools to add those artifacts to a copy of an offending (real but off message) stream and amplify the distribution of that fake-faked version with a debunking press release handy, so YMMV. While the Biden thing wasn't a deepfake directly, (shallow fake?) we're going to see tons of actual deepfakes around the election. IMO, elections are no longer between candidates, they are a war on truth for domination of the narrative - office is the effect. A campaign that focuses on what happens once the war is over is daydreaming about the future and distracted from the present and this will lose them key battles. For this reason, I think deepfakes are going to be the biggest weapon in campaign arsenals for the near future. Interesting times.

pantaloony 6y ago
I’m legitimately not sure democracy will survive modern, sophisticated propaganda techniques, plus an open, international Web, plus losing the ability to more-or-less trust audio and video recordings that we’ve grown used to over the last hundredish years. Between state actors and, eventually if not already, transnational corporations waging info warfare, I kinda doubt the institution can take it. Too much info, too fast, from too many sources.
- brutt 6y ago
  It looks like problem! :-) I have solution for this for years, but demand is not there yet.
  - ethbro 6y ago
    The solution is to make media literacy and logical fallacies a required course in democratic basic education.
    - 6y ago

yalogin 6y ago

There is already a large problem with political ads cherry picking and slicing audio and video to cheat viewers. I really worry that deep fakes will take it to another level completely. I fully expect the current administration to eagerly adopt it if available.

deleted 6y ago
[deleted]
acruns 6y ago
I just assume if a politician is speaking they are lying, at best misleading. Add in marketing, I assume it is a lie, regardless of political party.
- praptak 6y ago
  You still lose. There are players who don't need you to believe them, as it is sufficient for them if people cannot trust each other.
notahacker 6y ago
People have been impersonating politicians' accents pretty well for decades, mostly for comedy value, without tripping over sentence structures and pronunciation (unless that's the joke...) Other than lowering the effort bar, I'm not sure what deepfakes add. Sure, it means someone can make Donald Trump reading copypasta[1] or Dubya rapping [2] without knowing how to do impressions or even speaking the language natively, but a competent voice actor could do a much better job of putting words into politicians' mouths, and one thing political ads don't lack is budget. 1. https://www.youtube.com/watch?v=LEzIAixNkFI https://www.youtube.com/watch?v=LEzIAixNkFI 2. https://www.youtube.com/watch?v=LEzIAixNkFI https://www.youtube.com/watch?v=LEzIAixNkFI

paul7986 6y ago

Recently a friend changed her number and told me via text. Before adding her number i asked her a question that she and only I would know like who sat next to you at the old office. Think im going to keep doing this type of verification. It may annoy friends and family, but not sure how a hacker could ever know such small details between you and another.

Wistar 6y ago
Very good idea. A few years ago, I had a "friend" text me asking for emergency funds. It seemed wonky but within the realm of possibility. I asked a question about a mutual friend from the past revealed it as a scam.

leptoniscool 6y ago

How long does it take to train the model?

sidthekid 6y ago

The images of spectrogram analysis between the real and fake voices seemed to be distinguishable by the human eye. Can a image model be trained to detect fake voice spectrograms based on pitch and tone choppiness?

grogenaut 6y ago
Would you not just then take this and feed it into the training?
mensetmanusman 6y ago
The issue is that if you can measure it, you can train an AI to beat the other AI detecting it. As Pilate said: ‘Quid est veritas?‘
- deepnotderp 6y ago
  Generation is a much harder problem than discrimination though.

phjesusthatguy3 6y ago

Audio "deepfakes" have been worked on much longer than ones for video, although video deepfakes have the added issue of deep-faking synchronized audio. Today's consumers don't seem to be bothered by video deepfakes if they play to the beliefs of the audience.

inasio 6y ago

Next milestone I'm waiting for: Trump audio to Sarah Cooper video on the fly

eigenvalue 6y ago

My all-time favorite audio deepfake is Nobel prize winner Milton Friedman reading the lyrics to the 50 Cent track "PIMP". It really captures Friedman's tell-tale cadence and idiosyncratic lilt: https://www.youtube.com/watch?v=4mUYMvuNIas https://www.youtube.com/watch?v=4mUYMvuNIas

blueblisters 6y ago

There is an annual challenge for synthetic voice detection, ASVSpoof, that evaluates submissions on different types of attacks to speaker verification systems: text to speech, voice conversions and replayed attacks. The conclusion from the 2019 evaluation [1]: known synthetic deep fakes are fairly easy to detect using simple models with very low error rates (even high-fidelity techniques with Wavenet vocoders). [1]: ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection speech (https://www.isca-speech.org/archive/Interspeech_2019/pdfs/2249.pdf https://www.isca-speech.org/archive/Interspeech_2019/pdfs/22...)

kharak 6y ago

This might be paranoid. But I've established a protocol with some people in my life. Should someone with my voice ever contact them and ask for money (because emergency bla bla), nothing is to be done until a passphrase is mentioned. It's only a matter of time, until someone gets significant voice data and related contact numbers and proceeds with using those voices to train a model. Afterwards, that model will be used to real-time fake the original voice in a scamming attempt.

grumio 6y ago
That's good practice. Sadly these kinds of scams already happen without the effort of synthetic voices. Scammers call an older person and say "Hey Grandma, it's me your grandson. I need $500 for bail right away!" With the help of Facebook they can learn names and details to sound more convincing.