4 ms·

Phishing emails self select, I don't think generative ML would make a difference. Those "typos" are sometimes intentional. The scammers are not illiterate. They

Phishing emails self select, I don't think generative ML would make a difference. Those "typos" are sometimes intentional. The scammers are not illiterate. They are more than capable of using Grammarly if they wanted to.

brucethemoose2 3y ago
But what if emails can be mass produced at 15+ words/second, on a gaming PC? And what if they can very convincingly and quickly reply?
- 6177c40f 3y ago
  A bunch of random thoughts: Why would being able to write different emails help? Wouldn't they just reuse the same email over and over again, maybe just changing the addressee's name? Also, now I wonder what the typical response rate is for phishing emails? Are they already getting too many responses to keep up with or are they already managing just fine without bots? I think LLMs might be better for scams carried out over instant messaging services like Discord because that probably requires greater attention and flexibility on the part of the scammer. Still, has the number of people that could conceivably be scammed over an email or instant messaging-based scam increased? That is, will LLMs, by virtue of their response quality, responsiveness, or some other salient quality, increased the pool of people who could fall for these scams? If the answer is no, then I don't see how increased scale and throughput for human-quality text generation will really matter much.
  - _ea1k 3y ago
    My guess is that being able to generate lots of different text helps to bypass existing spam and phishing protections. Think of it as a large scale A/B test of circumvention techniques against both human and automated scam detection systems. Done right, it can effectively keep getting better on its own. Then add phone scamming and various other interactive scams once it is good enough. TBH, it is pretty scary where this will inevitably head.
- userbinator 3y ago
  And what if they can very convincingly and quickly reply? Turn this around, and have an LLM that replies to scammers and keeps them busy. There's already https://en.wikipedia.org/wiki/Lenny_(bot) https://en.wikipedia.org/wiki/Lenny_(bot) , but this is an area where AI could actually be a useful addition.
  - nwiswell 3y ago
    That gets pretty awkward the first time that a LLM turns out to have intentionally stalled a critical business process.
fnordpiglet 3y ago
My understanding, which can’t source the reference for off hand, is there is an idea that if someone can detect it’s a phishing attack from a typo they are too smart to fall for the overarching scam, hence the intentionality.
jokethrowaway 3y ago
I think there is some truth to that and some actual illiteracy going on

brucethemoose2 3y ago

But what if emails can be mass produced at 15+ words/second, on a gaming PC? And what if they can very convincingly and quickly reply?

6177c40f 3y ago
A bunch of random thoughts: Why would being able to write different emails help? Wouldn't they just reuse the same email over and over again, maybe just changing the addressee's name? Also, now I wonder what the typical response rate is for phishing emails? Are they already getting too many responses to keep up with or are they already managing just fine without bots? I think LLMs might be better for scams carried out over instant messaging services like Discord because that probably requires greater attention and flexibility on the part of the scammer. Still, has the number of people that could conceivably be scammed over an email or instant messaging-based scam increased? That is, will LLMs, by virtue of their response quality, responsiveness, or some other salient quality, increased the pool of people who could fall for these scams? If the answer is no, then I don't see how increased scale and throughput for human-quality text generation will really matter much.
- _ea1k 3y ago
  My guess is that being able to generate lots of different text helps to bypass existing spam and phishing protections. Think of it as a large scale A/B test of circumvention techniques against both human and automated scam detection systems. Done right, it can effectively keep getting better on its own. Then add phone scamming and various other interactive scams once it is good enough. TBH, it is pretty scary where this will inevitably head.
userbinator 3y ago
And what if they can very convincingly and quickly reply? Turn this around, and have an LLM that replies to scammers and keeps them busy. There's already https://en.wikipedia.org/wiki/Lenny_(bot) https://en.wikipedia.org/wiki/Lenny_(bot) , but this is an area where AI could actually be a useful addition.
- nwiswell 3y ago
  That gets pretty awkward the first time that a LLM turns out to have intentionally stalled a critical business process.

fnordpiglet 3y ago

My understanding, which can’t source the reference for off hand, is there is an idea that if someone can detect it’s a phishing attack from a typo they are too smart to fall for the overarching scam, hence the intentionality.

jokethrowaway 3y ago

I think there is some truth to that and some actual illiteracy going on