5 ms·

The Terrifying A.I. Scam That Uses Your Loved One's Voice

60 points by cromulent 3y ago

saliagato 3y ago
This sounds so fictionalized that may be fake
notTooFarGone 3y ago
Anyone believing AI will be a nice thing in the future: With data from social media it's a fucking nightmare - billions of people having their relationships visible to everyone and a singular voice recording/video is enough to scam your friends, family, workplace. I don't see a future of the current communication channels without strong authentication measures - people already get scammed with shitty tactics by billions of dollars per year and now you can automate it.
- lynx23 3y ago
  I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.
  - idank 3y ago
    It doesn't have to work 100% of the time to be a successful scam.
  - godelski 3y ago
    You're right to be suspicious but wrong in your interpretation of reality. The issue is that there's a big difference in how you'll interpret things in different environments and stress levels. Under pressure a lot of your normal checks will go out the window. This is why conmen often put you under time pressures, because they're stressing your system. Under normal circumstances, you're prefectly right, but under duress weird things seem normal. I mean duress is abnormal to begin with. Also enshitification is an important aspect here. We readily overlook things like sharp cuts in video, degraded signals in video and voice, distortion, etc. Because that stuff happens normally. Post hoc it is often easy to see scams and think how dumb someone must have been to fall for something so obvious, but that's actually a tool of the scammers themselves. They love that because the more embarrassed you are, the less likely you will report it due to shame. Be careful with over confidence, because that's what they target.
    - lynx23 3y ago
      As someone who works in IT for roughly 30 years, and who never fell for any kind of phishing, I beg to differ. I think my interpretation of reality, as you put it, isn't as bad as you might believe. Also, all the phishing-victims I know have something in common, a certain lack of rigour. You seem to claim that everyone can fall for phishing. I beg to differ, its a certain class of not very thorough (l)users. Your comment regarding enshitifcation has some truth to it I didn't really think about yet, thanks. However, my original comment still stands as it is. I challenge anyone to create a voice scam that fools me.
  - nickfromseattle 3y ago
    > that I'd be fooled by a fake call Can you say the same thing about all of your friends and loved ones? You don't have to personally fall for the scam for it to negatively impact you. Maybe your mom falls for it, and now she can't pay her mortgage. Fake call scams already generate billions of dollars in stolen money. Telecoms let spam phone calls run rampant for a decade before being forced to fix it. AI video/audio deepfakes at scale could 10x this industry.
  - beej71 3y ago
    There was an interesting paper about a year ago that talked about synthesizing voices with minimal training data, iirc on the order of 30 seconds. The results were imperfect but wholly impressive given the input. "Think of what will be possible just two more papers down the line."
  - Ferret7446 3y ago
    From experience, it is the people who are confident they won't be tricked that are most vulnerable to being tricked.
  - Turing_Machine 3y ago
    The spouse of a former boss of mine regularly mistook me for him, if I happened to answer his phone while he was at lunch or in the toilet or something (this was long before people had personal cell phones). Once she even said "Hi! This is your little snuggle bunny!". Mutual embarrassment ensued.
throwaway74432 3y ago
I recommend that everyone come up with a predetermined secret question that has both a real answer and a duress answer. Decide this question/answers with your loved ones in advance. If you're ever in a situation like this, you can ask the question to determine if its real and if they are being coerced.
atleastoptimal 3y ago
this is why i believe the only possible route forward for humanity in the AI age is a biometric digital footprint
- bboygravity 3y ago
  And a biometric digital footprint cannot be stolen or replicated by AI because?
  - atleastoptimal 3y ago
    Well it has to be something tied to the body, but also unique to the individual, like a fingerprint mixed with DNA
- godelski 3y ago
  Biological fingerprints are not that unique. Besides, how are you operating on it without digitizing it? What data cannot be replicated? The quantum no cloning theorem? Good luck getting that level of precision.
verisimi 3y ago
Well, I feel sorry for real kidnappers. Who's going to take them seriously now, when it is so easy to fake?! :) In all seriousness, the solution to this and other tech dystopia issues seems obvious to me. It's un-teching things. Having to go to the bank in person, etc. Not believing screens, even corporate or government ones. (We never should have believed those anyway, so this should be a net gain for humanity!)
godelski 3y ago
I wonder if these things will be the breaking point where we start to take security and identity verification seriously. I mean banks are still using SMS 2FA. Will tools like Keybase become well known and used? Yubi keys? Push into homomorphic encryption? Signal? Matrix? Don't get me started on SSNs... I think one of the unfortunate things about our societies is that we so strongly believe "if it ain't broke, don't fix it." It can be a useful piece of advice, but we should always be looking to improve things. Hell, it's what low level research is about. Just technically everything is broken, just how much. Similarly like "move fast and break things." It can be great but it leaves a wake of destruction and tech debt. Sometimes you go faster by fixing things first instead of hacking around things and compounding. How I think about it is that it is cheaper to perform maintenance rather than fixing things. Yet it is always easier to push off maintenance because other things are high priority. But in the long run this is far more costly. So we should often be fixing things before they are broken. But like all things, nothing is absolute ;)
- wreckdropibex 3y ago
  Identifying people online has already been solved https://news.ycombinator.com/item?id=39638744 https://news.ycombinator.com/item?id=39638744
djur 3y ago
This scam was being done years before AI voice cloning was possible (it's called a "grandparent scam") and so far I have not seen evidence that such technology is being used. The victims claim it was exactly like their relative's voice, but victims were saying the same thing two decades ago. The attorney mentioned in the article testified that his "son" was sobbing and claimed to have had his nose broken in the accident. Most people are not going to be able to confidently tell apart even a loved one's voice from someone around the same age if they're crying, screaming, being distorted by the phone transmission, etc. I think it's possible that scams like this could be done with voice cloning, don't get me wrong, but it's not necessary. It's probably good that people are alert to the risk of these scams, regardless.
scrps 3y ago
Use shibboleths. A friend and I have a few, not due specifically to cloning, mostly inside references, things only we would get. I could think of a few with family members I could use as well.
treme 3y ago
whatever happened to that yc company that was providing this service
- minimaxir 3y ago
  ElevenLabs (not YC), which powers most of the popular voice cloning controversies that make mainstream media, just raised a round at a valuation of $1.1B: https://www.reuters.com/technology/voice-ai-startup-elevenlabs-gains-unicorn-status-after-latest-fundraising-source-2024-01-22/ https://www.reuters.com/technology/voice-ai-startup-elevenla...
simultsop 3y ago
https://shorturl.at/iHPQW https://shorturl.at/iHPQW
shrubble 3y ago
It's right out of Terminator 2, except for the actually dead foster parents part.
- MrVandemar 3y ago
  It's right out of Terminator. The voice-altering scene in Terminator 2 was a replay of the original.
  - shrubble 3y ago
    You're right! Somehow the T2 scene stuck in my mind more...
- rKarpinski 3y ago
  What's wrong with Wolfie?
- blibble 3y ago
  OpenAI's military partners are likely working on that bit
xvector 3y ago
[flagged]
- neilkk 3y ago
  Good luck with that.
- kombookcha 3y ago
  It's insanely cruel and fucked up. Imagining what this is like for the victims goes way, way beyond what calling it a "scam" implies.
  - LadyCailin 3y ago
    It’s actual terrorism. It’s just small beans compared to other things which we usually call terrorism, but that’s what quite a large number of these “scams” are, small scale terrorism.
    - bboygravity 3y ago
      Ok, that's fine, but then we're going to need a new word to describe actual terrorism. This had/has a meaning that involves specifically targetting governments and destabilizing them.
      - xvector 3y ago
        Perhaps "psychological torture and extortion" also works as a phrase for this sort of crime.
    - deleted 3y ago
      [deleted]
  - nonrandomstring 3y ago
    And yet the top comment here is: "As horrible as this is, I suspect this will be far worse in enterprise companies." To give the "strongest interpretation" I presume by worse they mean more prevalent. But I also think a fundamental lack of human perspective in the tech world is the real root cause of cybersecurity problems threatening society. I mean, in this context who seriously cares about "enterprise" or money?
devjab 3y ago
As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to impersonate since there is often quite a lot of video footage freely available of them. You’re going to have to rely on personal confirmation, or digital signatures to do these sort of transfers. We use the digital signature since we have a system for that in Denmark, but we can’t use that for our Japanese department as an example as they don’t have a similar system which integrates with our national digital ID for companies. I guess our national system will eventually be extendable to Japanese citizens in some way, but it’ll likely never be extended to al the countries we operate in for various reasons. And this is us being sort of ahead of the curve on this, even for most companies around here in the worlds most digitalised country (unless that’s Estonia now). Interesting times.
- BOOSTERHIDROGEN 3y ago
  Interesting any article about the systems on denmark
  - piva00 3y ago
    It's more than Denmark, the Nordics in general use digital signatures for accessing banking, government services (tax agencies, healthcare, pension systems, etc.), online payment ID checks, and so on. In Denmark I believe NemID/MitID is the most used one, here in Sweden we have BankID (most used), and Freja eID, there are a few smaller ones as well; Norway has their own BankID that is independent from the Swedish one; Finland has their own as well (and I heard it's been live before the 2000s).
    - buzer 3y ago
      The original Finnish one was TUPAS, it has been available since at least 1998. It was phased out in 2019 as it was not compatible with eIDAS. For end users there isn't really any difference but the old system required separate contracts between each bank and service provider. Due to that service providers also couldn't use 3rd party (think service like Auth0) who would handle contracts with individual banks. From my understanding it was also fairly expensive (in terms of per authentication event). The new one is called Finnish Trust Network and uses OIDC and in some cases SAML, TUPAS had it's own protocol (I'm not sure if there even was a standard protocol for that kind of authentication back then, even SAML 1.0 came out only in 2002).

60 points by cromulent 3y ago

This sounds so fictionalized that may be fake

Anyone believing AI will be a nice thing in the future: With data from social media it's a fucking nightmare - billions of people having their relationships visible to everyone and a singular voice recording/video is enough to scam your friends, family, workplace. I don't see a future of the current communication channels without strong authentication measures - people already get scammed with shitty tactics by billions of dollars per year and now you can automate it.

lynx23 3y ago
I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.
- idank 3y ago
  It doesn't have to work 100% of the time to be a successful scam.
- godelski 3y ago
  You're right to be suspicious but wrong in your interpretation of reality. The issue is that there's a big difference in how you'll interpret things in different environments and stress levels. Under pressure a lot of your normal checks will go out the window. This is why conmen often put you under time pressures, because they're stressing your system. Under normal circumstances, you're prefectly right, but under duress weird things seem normal. I mean duress is abnormal to begin with. Also enshitification is an important aspect here. We readily overlook things like sharp cuts in video, degraded signals in video and voice, distortion, etc. Because that stuff happens normally. Post hoc it is often easy to see scams and think how dumb someone must have been to fall for something so obvious, but that's actually a tool of the scammers themselves. They love that because the more embarrassed you are, the less likely you will report it due to shame. Be careful with over confidence, because that's what they target.
  - lynx23 3y ago
    As someone who works in IT for roughly 30 years, and who never fell for any kind of phishing, I beg to differ. I think my interpretation of reality, as you put it, isn't as bad as you might believe. Also, all the phishing-victims I know have something in common, a certain lack of rigour. You seem to claim that everyone can fall for phishing. I beg to differ, its a certain class of not very thorough (l)users. Your comment regarding enshitifcation has some truth to it I didn't really think about yet, thanks. However, my original comment still stands as it is. I challenge anyone to create a voice scam that fools me.
- nickfromseattle 3y ago
  > that I'd be fooled by a fake call Can you say the same thing about all of your friends and loved ones? You don't have to personally fall for the scam for it to negatively impact you. Maybe your mom falls for it, and now she can't pay her mortgage. Fake call scams already generate billions of dollars in stolen money. Telecoms let spam phone calls run rampant for a decade before being forced to fix it. AI video/audio deepfakes at scale could 10x this industry.
- beej71 3y ago
  There was an interesting paper about a year ago that talked about synthesizing voices with minimal training data, iirc on the order of 30 seconds. The results were imperfect but wholly impressive given the input. "Think of what will be possible just two more papers down the line."
- Ferret7446 3y ago
  From experience, it is the people who are confident they won't be tricked that are most vulnerable to being tricked.
- Turing_Machine 3y ago
  The spouse of a former boss of mine regularly mistook me for him, if I happened to answer his phone while he was at lunch or in the toilet or something (this was long before people had personal cell phones). Once she even said "Hi! This is your little snuggle bunny!". Mutual embarrassment ensued.

throwaway74432 3y ago

I recommend that everyone come up with a predetermined secret question that has both a real answer and a duress answer. Decide this question/answers with your loved ones in advance. If you're ever in a situation like this, you can ask the question to determine if its real and if they are being coerced.

atleastoptimal 3y ago

this is why i believe the only possible route forward for humanity in the AI age is a biometric digital footprint

bboygravity 3y ago
And a biometric digital footprint cannot be stolen or replicated by AI because?
- atleastoptimal 3y ago
  Well it has to be something tied to the body, but also unique to the individual, like a fingerprint mixed with DNA
godelski 3y ago
Biological fingerprints are not that unique. Besides, how are you operating on it without digitizing it? What data cannot be replicated? The quantum no cloning theorem? Good luck getting that level of precision.

verisimi 3y ago

Well, I feel sorry for real kidnappers. Who's going to take them seriously now, when it is so easy to fake?! :) In all seriousness, the solution to this and other tech dystopia issues seems obvious to me. It's un-teching things. Having to go to the bank in person, etc. Not believing screens, even corporate or government ones. (We never should have believed those anyway, so this should be a net gain for humanity!)

godelski 3y ago

I wonder if these things will be the breaking point where we start to take security and identity verification seriously. I mean banks are still using SMS 2FA. Will tools like Keybase become well known and used? Yubi keys? Push into homomorphic encryption? Signal? Matrix? Don't get me started on SSNs... I think one of the unfortunate things about our societies is that we so strongly believe "if it ain't broke, don't fix it." It can be a useful piece of advice, but we should always be looking to improve things. Hell, it's what low level research is about. Just technically everything is broken, just how much. Similarly like "move fast and break things." It can be great but it leaves a wake of destruction and tech debt. Sometimes you go faster by fixing things first instead of hacking around things and compounding. How I think about it is that it is cheaper to perform maintenance rather than fixing things. Yet it is always easier to push off maintenance because other things are high priority. But in the long run this is far more costly. So we should often be fixing things before they are broken. But like all things, nothing is absolute ;)

wreckdropibex 3y ago
Identifying people online has already been solved https://news.ycombinator.com/item?id=39638744 https://news.ycombinator.com/item?id=39638744

djur 3y ago

This scam was being done years before AI voice cloning was possible (it's called a "grandparent scam") and so far I have not seen evidence that such technology is being used. The victims claim it was exactly like their relative's voice, but victims were saying the same thing two decades ago. The attorney mentioned in the article testified that his "son" was sobbing and claimed to have had his nose broken in the accident. Most people are not going to be able to confidently tell apart even a loved one's voice from someone around the same age if they're crying, screaming, being distorted by the phone transmission, etc. I think it's possible that scams like this could be done with voice cloning, don't get me wrong, but it's not necessary. It's probably good that people are alert to the risk of these scams, regardless.

scrps 3y ago

Use shibboleths. A friend and I have a few, not due specifically to cloning, mostly inside references, things only we would get. I could think of a few with family members I could use as well.

treme 3y ago

whatever happened to that yc company that was providing this service

minimaxir 3y ago
ElevenLabs (not YC), which powers most of the popular voice cloning controversies that make mainstream media, just raised a round at a valuation of $1.1B: https://www.reuters.com/technology/voice-ai-startup-elevenlabs-gains-unicorn-status-after-latest-fundraising-source-2024-01-22/ https://www.reuters.com/technology/voice-ai-startup-elevenla...

simultsop 3y ago

https://shorturl.at/iHPQW https://shorturl.at/iHPQW

shrubble 3y ago

It's right out of Terminator 2, except for the actually dead foster parents part.

MrVandemar 3y ago
It's right out of Terminator. The voice-altering scene in Terminator 2 was a replay of the original.
- shrubble 3y ago
  You're right! Somehow the T2 scene stuck in my mind more...
rKarpinski 3y ago
What's wrong with Wolfie?
blibble 3y ago
OpenAI's military partners are likely working on that bit

xvector 3y ago

[flagged]

neilkk 3y ago
Good luck with that.
kombookcha 3y ago
It's insanely cruel and fucked up. Imagining what this is like for the victims goes way, way beyond what calling it a "scam" implies.
- LadyCailin 3y ago
  It’s actual terrorism. It’s just small beans compared to other things which we usually call terrorism, but that’s what quite a large number of these “scams” are, small scale terrorism.
  - bboygravity 3y ago
    Ok, that's fine, but then we're going to need a new word to describe actual terrorism. This had/has a meaning that involves specifically targetting governments and destabilizing them.
    - xvector 3y ago
      Perhaps "psychological torture and extortion" also works as a phrase for this sort of crime.
  - deleted 3y ago
    [deleted]
- nonrandomstring 3y ago
  And yet the top comment here is: "As horrible as this is, I suspect this will be far worse in enterprise companies." To give the "strongest interpretation" I presume by worse they mean more prevalent. But I also think a fundamental lack of human perspective in the tech world is the real root cause of cybersecurity problems threatening society. I mean, in this context who seriously cares about "enterprise" or money?

devjab 3y ago

As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to impersonate since there is often quite a lot of video footage freely available of them. You’re going to have to rely on personal confirmation, or digital signatures to do these sort of transfers. We use the digital signature since we have a system for that in Denmark, but we can’t use that for our Japanese department as an example as they don’t have a similar system which integrates with our national digital ID for companies. I guess our national system will eventually be extendable to Japanese citizens in some way, but it’ll likely never be extended to al the countries we operate in for various reasons. And this is us being sort of ahead of the curve on this, even for most companies around here in the worlds most digitalised country (unless that’s Estonia now). Interesting times.

BOOSTERHIDROGEN 3y ago
Interesting any article about the systems on denmark
- piva00 3y ago
  It's more than Denmark, the Nordics in general use digital signatures for accessing banking, government services (tax agencies, healthcare, pension systems, etc.), online payment ID checks, and so on. In Denmark I believe NemID/MitID is the most used one, here in Sweden we have BankID (most used), and Freja eID, there are a few smaller ones as well; Norway has their own BankID that is independent from the Swedish one; Finland has their own as well (and I heard it's been live before the 2000s).
  - buzer 3y ago
    The original Finnish one was TUPAS, it has been available since at least 1998. It was phased out in 2019 as it was not compatible with eIDAS. For end users there isn't really any difference but the old system required separate contracts between each bank and service provider. Due to that service providers also couldn't use 3rd party (think service like Auth0) who would handle contracts with individual banks. From my understanding it was also fairly expensive (in terms of per authentication event). The new one is called Finnish Trust Network and uses OIDC and in some cases SAML, TUPAS had it's own protocol (I'm not sure if there even was a standard protocol for that kind of authentication back then, even SAML 1.0 came out only in 2002).