3 ms·

I think the answer, counterintuitively, is to make these AI tools more open and accessible. As long as they're restricted or regulated or inaccessible people wi

I think the answer, counterintuitively, is to make these AI tools more open and accessible. As long as they're restricted or regulated or inaccessible people will continue to think of videos and recordings as not fakeable. But make voice cloning something easy and fun to do with a $1 app, let the teens have their prank call fun and pretty soon it should work its way into the public consciousness. I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. I think it might be a rough couple years but hopefully we'll be through it soon.

CoastalCoder 2y ago
> I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?
- alternatex 2y ago
  Skype does it in real time (for live translation) with a few seconds of audio. For the reasons discussed in this thread, it continuously forgets its previous training from the call to not make the voice too similar, but just similar enough to distinguish the speakers.
- jerpint 2y ago
  Just a few seconds of speech gets you pretty far with a lot of models
  - caeril 2y ago
    Yes, this is generally true, in order to clone the timbre/pitch/tonality, but if you're going to run a voice cloning scam to fool friends and family members, you need much more to get the cadence, pauses, vocal tics, etc right.
- HeatrayEnjoyer 2y ago
  Almost none, at most as little as a professional impersonator requires. GPT-4o's advanced voice mode would clone the user's voice by accident. A recording clip of one incident is available: >During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice. >Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7ZroN/28ddb86e84776d751c8edea4cdb57873/voice_generation1.wav https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7... https://openai.com/index/gpt-4o-system-card/ https://openai.com/index/gpt-4o-system-card/
HeatrayEnjoyer 2y ago
This is idealistic. People still haven't fully learned that images can be photoshopped in its twenty years of its existence. (Deep)faked porn is still harmful which is why it's a crime. Worse, there isn't an attitude of default skepticism in many areas/cultures. If a person is suspected of violating the moral code the priority will be punishment and reinforcing that such behavior isn't acceptable. Whether or not the specific person actually did the specific act is a secondary concern. It's just going to increase the number of people who will be harmed or killed.
- deepsun 2y ago
  Yep, had a lawyer telling me that image timestamp cannot be faked. While it's literally a right-click away.
  - veunes 2y ago
    When people in positions of authority, like legal experts, don’t fully grasp how easily digital content can be manipulated
  - sugarkjube 2y ago
    Well, thing is, most people can't. In a lawyers view, and a judge's view, some skilled expert "hackers" can, and its called hacking. (so i guess we're all hackers) I once discussed these things with a (knowledgeable) lawyer. He explained you can just present almost anhthing in a court case, and when it isn't refuted, well then it's valid. In a case my lawyer (same one) presented a printed out email. Other party did not claim it was false, so it's suddenly just as valid as a registered letter. (it was a genuine email). In another unrelated case, the other party suddenly introduced a forged picture. If I hadn't been there at that moment (I wasn't supposed to actually), then suddenly it would have been proof. Court cases are not about truth, and not about justice. They are about convincing the judge.
    - deepsun 2y ago
      Well, at least for email, it's theoretically possible to prove it's authenticity through third-parties. E.g. lawyer can ask GMail "did you receive this email with this DKIM?".
bryanlarsen 2y ago
Most people will believe a rumour if it is told to them in person by a friend. We've had our entire evolution worth of time to recognize that rumours can be manipulated yet rumours still spread and are still very dangerous.

CoastalCoder 2y ago

> I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something. Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?

alternatex 2y ago
Skype does it in real time (for live translation) with a few seconds of audio. For the reasons discussed in this thread, it continuously forgets its previous training from the call to not make the voice too similar, but just similar enough to distinguish the speakers.
jerpint 2y ago
Just a few seconds of speech gets you pretty far with a lot of models
- caeril 2y ago
  Yes, this is generally true, in order to clone the timbre/pitch/tonality, but if you're going to run a voice cloning scam to fool friends and family members, you need much more to get the cadence, pauses, vocal tics, etc right.
HeatrayEnjoyer 2y ago
Almost none, at most as little as a professional impersonator requires. GPT-4o's advanced voice mode would clone the user's voice by accident. A recording clip of one incident is available: >During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice. >Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7ZroN/28ddb86e84776d751c8edea4cdb57873/voice_generation1.wav https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7... https://openai.com/index/gpt-4o-system-card/ https://openai.com/index/gpt-4o-system-card/

HeatrayEnjoyer 2y ago

This is idealistic. People still haven't fully learned that images can be photoshopped in its twenty years of its existence. (Deep)faked porn is still harmful which is why it's a crime. Worse, there isn't an attitude of default skepticism in many areas/cultures. If a person is suspected of violating the moral code the priority will be punishment and reinforcing that such behavior isn't acceptable. Whether or not the specific person actually did the specific act is a secondary concern. It's just going to increase the number of people who will be harmed or killed.

deepsun 2y ago
Yep, had a lawyer telling me that image timestamp cannot be faked. While it's literally a right-click away.
- veunes 2y ago
  When people in positions of authority, like legal experts, don’t fully grasp how easily digital content can be manipulated
- sugarkjube 2y ago
  Well, thing is, most people can't. In a lawyers view, and a judge's view, some skilled expert "hackers" can, and its called hacking. (so i guess we're all hackers) I once discussed these things with a (knowledgeable) lawyer. He explained you can just present almost anhthing in a court case, and when it isn't refuted, well then it's valid. In a case my lawyer (same one) presented a printed out email. Other party did not claim it was false, so it's suddenly just as valid as a registered letter. (it was a genuine email). In another unrelated case, the other party suddenly introduced a forged picture. If I hadn't been there at that moment (I wasn't supposed to actually), then suddenly it would have been proof. Court cases are not about truth, and not about justice. They are about convincing the judge.
  - deepsun 2y ago
    Well, at least for email, it's theoretically possible to prove it's authenticity through third-parties. E.g. lawyer can ask GMail "did you receive this email with this DKIM?".

bryanlarsen 2y ago

Most people will believe a rumour if it is told to them in person by a friend. We've had our entire evolution worth of time to recognize that rumours can be manipulated yet rumours still spread and are still very dangerous.