5 ms·

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Lla

by dbspin 2y ago

I'm kind of amazed how slow 'AI phishing' has been to roll out. The technology for customised text based attacks at scale has been available at least since Llama was open sourced. The tech for custom voice and image based attacks is basically there too with whisper / tortoise and stable diffusion - though clearly more expensive to render. I'm honestly not sure why social networks aren't being leveraged more to target and spoof individuals - especially elderly people. Tailored attacks impersonating text or voice messages from close contacts and family members should be fairly common, and yet they're not. Robo-calls that carry out a two way conversation convincingly impersonating bank or police officials should be everywhere. Yet the only spam-calls I ever receive are from Indian call centres or static messages using decades old synthesised voice tech.

Der_Einzige 2y ago
It appears that criminals really are stupid, and thank goodness for it!
- cess11 2y ago
  Most people are criminals. Speeding, piracy, dubious porn, and so on. In a wider sense consumption of products or other use of criminally exploited labour. At the same time, most people are more clever than one tends to expect.
  - throwup238 2y ago
    ”If the prosecutor is obliged to choose his cases, it follows that he can choose his defendants. Therein is the most dangerous power of the prosecutor: that he will pick people that he thinks he should get, rather than pick cases that need to be prosecuted. With the law books filled with a great assortment of crimes, a prosecutor stands a fair chance of finding at least a technical violation of some act on the part of almost anyone." -Justice Robert H Jackson
- ToucanLoucan 2y ago
  Why bother operating a much more complex LLM stack when you're already raking in cash from confused boomers trying to pay the IRS off with iTunes gift cards? Their system works. They'll take up machine learning powered tools once the old farts all die off/go broke and they need more complicated scams for more technology-savvy victims. I'm being glib here but also if you're the type of person who gets texts from the IRS from a number you've never seen and take it at face value that you can pay off your overdue tax bill with gift cards... like, you are already the perfect victim for this sort of scam. They don't need to be good, they just need you to self identify and leap right into the trap.
jjulius 2y ago
Forgive my ignorance, but why are we surprised that voice messages aren't being spoofed more often? Doesn't this require a pretty darn decent dataset for training? Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing.
- savanaly 2y ago
  The dataset you need to train in the first place is indeed huge, but I think the idea is once them model is trained, new "voices" can be acquired with much less data than was required to train it in the first place. Just like you can instruct ChatGPT to talk about topics never heard of on the internet and in a dialect you customize and invent on the spot and it can comply, despite not consuming an internet's worth of subject matter about it.
  - nanidin 2y ago
    Soon the role of the Indian call centers will change from running the scam directly to making spam calls to trusted contacts of the intended mark to collect voice data for TTS model fine tuning.
- Zambyte 2y ago
  > Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing. That sounds like a great recipe for spear phishing public figures.
- dbspin 2y ago
  I'm not a data science / AI person at all, but AFAIK while initial model training is enormously processor intensive, customising a trained model is not. This has been my experience playing with custom training stable diffusion on very low memory home GPU. I'd assume it's true for voice generators like Tortoise [1] also. Moreover, while Eleven Labs isn't in my opinion as good, they let you custom train voices for an incredibly low cost and with a tiny amount of sample data [3]. For perfect audio spoofing, lots of audio would be needed. Bear in mind there are literally millions of podcasts available [2], and billions of youtube videos. Should be trivial to grab biographic data and voice samples from a subset of them. [1] https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts [2] https://whatsthebigdata.com/podcasts-statistics/ https://whatsthebigdata.com/podcasts-statistics/ [3] https://elevenlabs.io/ https://elevenlabs.io/
diatone 2y ago
It’s not enough for the method to be possible, it has to be economical. If it’s not prevalent yet it means one of two things: (1) cheaper methods work too well for now to even bother with sophisticated approaches (2) it’s too expensive to be worth the effort It would be interesting to plot out the cost of an integrated AI stack for this over time.
- tijtij 2y ago
  >(1) cheaper methods work too well for now to even bother with sophisticated approaches Pig butchering scams uses all forms of kidnapping, human trafficking, and slave labor
- dbspin 2y ago
  There's a third possibility. It's just not turnkey enough yet for criminal enterprises to bother. I'm sure cartels and mafia level gangsters sometimes have great tech people, but the level of op sec and technical knowhow exhibited by most professional criminals seems low. https://www.npr.org/2024/05/31/1197959218/fbi-phone-company-anom https://www.npr.org/2024/05/31/1197959218/fbi-phone-company-...
kkielhofner 2y ago
The ROI on scams based in Indian call centers is already huge. They recently took my mom for $25k for what was a few hours of “work” over the span of two days. When I reviewed their communications and got the full story from my mom they’re in some ways laughably bad and in other ways very cunning. Turns out it all started with a comically bad initial e-mail, pop-up, and then remote access. Then follow-up calls and text messages (Bitcoin QR code with her bank logo) and multiple people impersonating banks, various government agencies, etc. The end to end pipeline to replicate this via AI would be very complex and difficult (today). I imagine the increased scale, additional opportunity, and reduced “payroll” that could be realized utilizing AI (given the initial level of development effort) just isn’t there. Yet.
- dbspin 2y ago
  I'm so sorry, that's absolutely awful. Hope your moms OK.
- burningChrome 2y ago
  Banks are actually getting much better at ferreting out these scams. My Mom got one of those calls where the guy said my son was jail and they needed to pay his cash bail to get him out. They never called me or him, just panicked and went to the bank to get the cash out and the bank refused to give them the money because they knew it was a scam. They started asking my parents questions and pointing out how the scam worked when they got agitated they weren't giving them the money. They called me from the bank and I texted my son and he was completely confused since he was at work and was just fine. I thanked the people at the bank profusely for their intervention. It was a great learning experience for my parents as well. They are way more leery about strangers calling and have already hung up on several scammers. They also filter out email messages and won't click on any links in emails or pop-ups. The whole experience really put their radar up for this stuff now.
  - angry_old_man_5 2y ago
    I've read that even Target does this now, especially when an elderly person comes in trying to buy thousands of dollars worth of gift cards. Unfortunately too late for my grandmother. In her case, she'll know it's a scam and still participate just because she's bored.

by dbspin 2y ago

Der_Einzige 2y ago

It appears that criminals really are stupid, and thank goodness for it!

cess11 2y ago
Most people are criminals. Speeding, piracy, dubious porn, and so on. In a wider sense consumption of products or other use of criminally exploited labour. At the same time, most people are more clever than one tends to expect.
- throwup238 2y ago
  ”If the prosecutor is obliged to choose his cases, it follows that he can choose his defendants. Therein is the most dangerous power of the prosecutor: that he will pick people that he thinks he should get, rather than pick cases that need to be prosecuted. With the law books filled with a great assortment of crimes, a prosecutor stands a fair chance of finding at least a technical violation of some act on the part of almost anyone." -Justice Robert H Jackson
ToucanLoucan 2y ago
Why bother operating a much more complex LLM stack when you're already raking in cash from confused boomers trying to pay the IRS off with iTunes gift cards? Their system works. They'll take up machine learning powered tools once the old farts all die off/go broke and they need more complicated scams for more technology-savvy victims. I'm being glib here but also if you're the type of person who gets texts from the IRS from a number you've never seen and take it at face value that you can pay off your overdue tax bill with gift cards... like, you are already the perfect victim for this sort of scam. They don't need to be good, they just need you to self identify and leap right into the trap.

jjulius 2y ago

Forgive my ignorance, but why are we surprised that voice messages aren't being spoofed more often? Doesn't this require a pretty darn decent dataset for training? Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing.

savanaly 2y ago
The dataset you need to train in the first place is indeed huge, but I think the idea is once them model is trained, new "voices" can be acquired with much less data than was required to train it in the first place. Just like you can instruct ChatGPT to talk about topics never heard of on the internet and in a dialect you customize and invent on the spot and it can comply, despite not consuming an internet's worth of subject matter about it.
- nanidin 2y ago
  Soon the role of the Indian call centers will change from running the scam directly to making spam calls to trusted contacts of the intended mark to collect voice data for TTS model fine tuning.
Zambyte 2y ago
> Unless they've got a ton of videos of themselves shared on a public social media profile, I don't know that this is going to be a thing. That sounds like a great recipe for spear phishing public figures.
dbspin 2y ago
I'm not a data science / AI person at all, but AFAIK while initial model training is enormously processor intensive, customising a trained model is not. This has been my experience playing with custom training stable diffusion on very low memory home GPU. I'd assume it's true for voice generators like Tortoise [1] also. Moreover, while Eleven Labs isn't in my opinion as good, they let you custom train voices for an incredibly low cost and with a tiny amount of sample data [3]. For perfect audio spoofing, lots of audio would be needed. Bear in mind there are literally millions of podcasts available [2], and billions of youtube videos. Should be trivial to grab biographic data and voice samples from a subset of them. [1] https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts [2] https://whatsthebigdata.com/podcasts-statistics/ https://whatsthebigdata.com/podcasts-statistics/ [3] https://elevenlabs.io/ https://elevenlabs.io/

diatone 2y ago

It’s not enough for the method to be possible, it has to be economical. If it’s not prevalent yet it means one of two things: (1) cheaper methods work too well for now to even bother with sophisticated approaches (2) it’s too expensive to be worth the effort It would be interesting to plot out the cost of an integrated AI stack for this over time.

tijtij 2y ago
>(1) cheaper methods work too well for now to even bother with sophisticated approaches Pig butchering scams uses all forms of kidnapping, human trafficking, and slave labor
dbspin 2y ago
There's a third possibility. It's just not turnkey enough yet for criminal enterprises to bother. I'm sure cartels and mafia level gangsters sometimes have great tech people, but the level of op sec and technical knowhow exhibited by most professional criminals seems low. https://www.npr.org/2024/05/31/1197959218/fbi-phone-company-anom https://www.npr.org/2024/05/31/1197959218/fbi-phone-company-...

kkielhofner 2y ago

The ROI on scams based in Indian call centers is already huge. They recently took my mom for $25k for what was a few hours of “work” over the span of two days. When I reviewed their communications and got the full story from my mom they’re in some ways laughably bad and in other ways very cunning. Turns out it all started with a comically bad initial e-mail, pop-up, and then remote access. Then follow-up calls and text messages (Bitcoin QR code with her bank logo) and multiple people impersonating banks, various government agencies, etc. The end to end pipeline to replicate this via AI would be very complex and difficult (today). I imagine the increased scale, additional opportunity, and reduced “payroll” that could be realized utilizing AI (given the initial level of development effort) just isn’t there. Yet.

dbspin 2y ago
I'm so sorry, that's absolutely awful. Hope your moms OK.
burningChrome 2y ago
Banks are actually getting much better at ferreting out these scams. My Mom got one of those calls where the guy said my son was jail and they needed to pay his cash bail to get him out. They never called me or him, just panicked and went to the bank to get the cash out and the bank refused to give them the money because they knew it was a scam. They started asking my parents questions and pointing out how the scam worked when they got agitated they weren't giving them the money. They called me from the bank and I texted my son and he was completely confused since he was at work and was just fine. I thanked the people at the bank profusely for their intervention. It was a great learning experience for my parents as well. They are way more leery about strangers calling and have already hung up on several scammers. They also filter out email messages and won't click on any links in emails or pop-ups. The whole experience really put their radar up for this stuff now.
- angry_old_man_5 2y ago
  I've read that even Target does this now, especially when an elderly person comes in trying to buy thousands of dollars worth of gift cards. Unfortunately too late for my grandmother. In her case, she'll know it's a scam and still participate just because she's bored.