3 ms·

This lines up well with the success rates I have seen from expert phishers. When I worked at a certain well known company with strong security, a demon called K

This lines up well with the success rates I have seen from expert phishers. When I worked at a certain well known company with strong security, a demon called Karla would succeed at spearphishing a bit over 50% of the security team. AI now means much less skilled people can be as good as she was. Karla as a Service. We are doomed.

richdougherty 2y ago
"The cost-effective nature of AI makes it highly plausible we're moving towards an agent vs agent future." Sounds right. I assume we will all have AI agents triaging our emails trying to protect us. Maybe we will need AI to help us discern what is really true when we search for or consume information as well. The amount and quality of plausible but fake information is only going to increase. "However, the possibilities of jailbreaks and prompt injections pose a significant challenge to using language models to prevent phishing." Gives a hint at the arms race between attack and defense.
- throw10920 2y ago
  I don't think that there will necessarily be an arms race. Some security problems are deterministically solvable and don't need AI. For instance, there is a very good classical algorithm for preventing password brute-forcing - exponential backoff on failure per IP address, maybe with some additional per-account backoff as well. Combined with sane password rules (e.g. correct horse battery staple, not "you must have one character from every language in Madagascar), make password brute-forcing infeasible, and force attackers to try other approaches - which in the security world counts as success. No AI needed.
ninkendo 2y ago
What defines a successful spear phishing? Is it just clicking a link? My process when I see a sketchy email is to hover over the links to see the domain. Phishing links are obvious to anyone who understands how URLs and DNS works. But working for a typical enterprise, all links are “helpfully” rewritten to some dumbass phishing detection service, so I can no longer do this. At my current company I got what I assumed was a phishing email, I hovered over the links, saw they were pointing to some dipshit outlook phishing detection domain, and decided “what the hell, may as well click… may as well see if this phishing detection flags it” [0]… … and it turns out it was not only not legit, but it was an internal phishing test email to see whether I’d “fall for” a phishing link. Note that the test didn’t check if I’d, say, enter my credentials into a fraudulent website. It considered me to have failed if I merely clicked a link. A link to our internal phishing detection service because of course I’m not trusted to see the actual link itself (because I’d use that to check the DNS name.) I guess the threat model is that these phishers have a zero-day browser vulnerability (worth millions on auction sites) and that I’d be instantly owned the moment I clicked an outlook phishing service link, so I failed that. Also note that this was a “spear phishing” email, so it looked like any normal internal company email (in this case to a confluence page) and had my name on it. So given that it looks nearly identical to other corporate emails, and that you can’t actually see the links (they’re all rewritten), the takeaway is that you simply cannot use email to click links, ever, in a modern company with typical infosec standards. Ever ever. Zero exceptions. - [0] My threat model doesn’t include “malware installed the moment I click a link, on an up to date browser”, because I don’t believe spear phishers have those sort of vulnerabilities available to burn, given the millions of dollars that costs.
- bongodongobob 2y ago
  Problem is Outlook now obfuscates the shit out of links, something something safesearch or along those lines. When I hover over a link, I now have no idea where it wants to take me unless I copy and paste it and look through the 500 character link to find where it actually wants to take me.
- alex989 2y ago
  It gets worse, if you try to see the url from a phone, there is a good chance it will load the page for you to show a preview. They think it's helpful.... that motherfucking preview on the iPhone forced me to spend a full hour in training because they think I clicked on the link. Now I send all of these types of email to spam and don't give a fuck. Anything "internal" with a link to click goes to spam unless it's directly from my boss. Turns out 99% of it is not that important.
  - deleted 2y ago
    [deleted]
DoctorOetker 2y ago
Is this the same Karla as in Fight Club?
- pavel_lishin 2y ago
  I believe that was Marla.
  - DoctorOetker 2y ago
    you are correct, not sure if this is the origin story from before she changed her name.
edm0nd 2y ago
This story is extra hilarious to me because a threat actor has made a service named karla.pw that lets you harvest info from infostealers and creds dumps.

richdougherty 2y ago

"The cost-effective nature of AI makes it highly plausible we're moving towards an agent vs agent future." Sounds right. I assume we will all have AI agents triaging our emails trying to protect us. Maybe we will need AI to help us discern what is really true when we search for or consume information as well. The amount and quality of plausible but fake information is only going to increase. "However, the possibilities of jailbreaks and prompt injections pose a significant challenge to using language models to prevent phishing." Gives a hint at the arms race between attack and defense.

throw10920 2y ago
I don't think that there will necessarily be an arms race. Some security problems are deterministically solvable and don't need AI. For instance, there is a very good classical algorithm for preventing password brute-forcing - exponential backoff on failure per IP address, maybe with some additional per-account backoff as well. Combined with sane password rules (e.g. correct horse battery staple, not "you must have one character from every language in Madagascar), make password brute-forcing infeasible, and force attackers to try other approaches - which in the security world counts as success. No AI needed.

ninkendo 2y ago

What defines a successful spear phishing? Is it just clicking a link? My process when I see a sketchy email is to hover over the links to see the domain. Phishing links are obvious to anyone who understands how URLs and DNS works. But working for a typical enterprise, all links are “helpfully” rewritten to some dumbass phishing detection service, so I can no longer do this. At my current company I got what I assumed was a phishing email, I hovered over the links, saw they were pointing to some dipshit outlook phishing detection domain, and decided “what the hell, may as well click… may as well see if this phishing detection flags it” [0]… … and it turns out it was not only not legit, but it was an internal phishing test email to see whether I’d “fall for” a phishing link. Note that the test didn’t check if I’d, say, enter my credentials into a fraudulent website. It considered me to have failed if I merely clicked a link. A link to our internal phishing detection service because of course I’m not trusted to see the actual link itself (because I’d use that to check the DNS name.) I guess the threat model is that these phishers have a zero-day browser vulnerability (worth millions on auction sites) and that I’d be instantly owned the moment I clicked an outlook phishing service link, so I failed that. Also note that this was a “spear phishing” email, so it looked like any normal internal company email (in this case to a confluence page) and had my name on it. So given that it looks nearly identical to other corporate emails, and that you can’t actually see the links (they’re all rewritten), the takeaway is that you simply cannot use email to click links, ever, in a modern company with typical infosec standards. Ever ever. Zero exceptions. - [0] My threat model doesn’t include “malware installed the moment I click a link, on an up to date browser”, because I don’t believe spear phishers have those sort of vulnerabilities available to burn, given the millions of dollars that costs.

bongodongobob 2y ago
Problem is Outlook now obfuscates the shit out of links, something something safesearch or along those lines. When I hover over a link, I now have no idea where it wants to take me unless I copy and paste it and look through the 500 character link to find where it actually wants to take me.
alex989 2y ago
It gets worse, if you try to see the url from a phone, there is a good chance it will load the page for you to show a preview. They think it's helpful.... that motherfucking preview on the iPhone forced me to spend a full hour in training because they think I clicked on the link. Now I send all of these types of email to spam and don't give a fuck. Anything "internal" with a link to click goes to spam unless it's directly from my boss. Turns out 99% of it is not that important.
- deleted 2y ago
  [deleted]

DoctorOetker 2y ago

Is this the same Karla as in Fight Club?

pavel_lishin 2y ago
I believe that was Marla.
- DoctorOetker 2y ago
  you are correct, not sure if this is the origin story from before she changed her name.

edm0nd 2y ago

This story is extra hilarious to me because a threat actor has made a service named karla.pw that lets you harvest info from infostealers and creds dumps.