13 ms·
Human study on AI spear phishing campaigns
- TechDebtDevin 2y agoGrandma is fkd
- noman-land 2y agoWe are all grandma.
- pavel_lishin 2y agoI've had coworkers, and so has my spouse, who has fallen for the "iTunes gift cards for the CEO" trick. I think grandma is no longer an accurate stand-in for a tech-unsavvy person who is vulnerable to spearphishing attempts.
- consumer451 2y agoI would argue that pretty much everyone could be socially engineered into dropping their guard for a moment.
- e40 2y agoI get about 10 emails per week from my "CEO" to pay an invoice. I've even gotten a few text messages. Oddly, the emails never have an attachment. Is this because Google (Workspace account) is removing it? I've always wondered if it is 10 different orgs doing the campaigns, or the same one. If the same one, why send 10?
- noman-land 2y agoThis is somehow not considered to be an active warzone when it clearly is. The slightest misstep could ruin your life.
- pavel_lishin 2y ago> I've always wondered if it is 10 different orgs doing the campaigns, or the same one. If the same one, why send 10? My bet is that one criminal group is selling software to enable this, with very similar default settings. Then ten groups by the software, and each one ends up sending you a very similar email.
- terribleperson 2y agoThis is one of the terrifying, probably already happening threats presented by current LLMs. Social engineering (and I include spearphishing) has always been powerful and hard to mitigate. Now it can be done automatically at low cost.
- cluckindan 2y agoIf the study was done with target consent, it might be biased with inflated click-through rates due to the targets expecting benign well-targeted spear-phishing messages. If it was done without target consent, it would certainly be unethical.
- neom 2y agoThey got IRB approval. The authors framed the emails as part of a marketing study involving "targeted marketing emails."
- jt2190 2y agoIt seems like “the subject clicked a link in an email” is equated to “being phished”, but I’m not certain that is a good definition.
- Terr_ 2y agoI'm certain that someday I'm going to be dinged on a really shallow kind of work security test because I decided to investigate a link into a sandbox/honeypot environment.
- RaptorJ 2y agoThese phish testing companies always stick a header (X-PHISH-TEST or some such) on the email so the email server can white-list -- easy to just Outlook blackhole filter anything with that header after you've seen one test.
- Terr_ 2y agoWhat stops an attacker from abusing the same header? It could be kinda-secure if the header had to have a payload which matched a certain value pre-approved for a time-period. However an insider threat could see the test going on and then launch their own campaign during the validity window.
- joe_the_user 2y agoThat, after thirty years, email security still depends on the wisdom of individuals not clicking the wrong link, is appalling. The situation involves institutions happy to opaque links to email as part of their workflow. What could change this? All I can imagine is state regulation but that also is implausible.
- concerndc1tizen 2y agoThe same is true for operating systems. Why don't they sandbox properly? We have sandboxing on mobile apps. Why can't we have the same for desktop?
- andersa 2y agoAfter all these years, Microsoft is finally rolling out win32 app isolation, so maybe we are finally on the good path...
- fassssst 2y agoDevelopers initially revolted against Microsoft UWP and Mac App Store.
- wetpaws 2y agoWalled garden is not a substitute for security
- wongarsu 2y agoAPPX (the installer format used by the windows store) and its successor MSIX contain decent security improvements, including filesystem and registry virtualisation and a capability-based permission system. After the limited success of the windows store you can now get the same in standalone installers. It has been adopted by approximately nobody
- Y_Y 2y agoNot because they isolated the applications though! Because they were shit, and that's not a requirement.
- LeftHandPath 2y agoThey built their phishing emails using data scraped from public profiles. Fascinating. I have to wonder if, in the near future, we're going to have a much higher perceived cost for online social media usage. Problems we're already seeing: - AI turning clothed photos into the opposite [0] - AI mimicking a person's voice, given enough reference material [1] - Scammers impersonating software engineers in job interviews, after viewing their LinkedIn or GitHub profiles [2] - Fraudsters using hacked GitHub accounts to trick other developers into downloading/cloning malicious arbitrary code [3] - AI training on publicly-available text, photo, and video, to the surprise of content creators (but arguably fair use) [4] - AI spamming github issues to try to claim bug bounties [5] All of this probably sounds like a "well, duh" to some of the more privacy and security savvy here, but I still think it has created a notable shift from the tech-optimism that ran from 2012-2018 or so. These problems all existed then, too, but with less frequency. Now, it's a full-pressure firehose. [0]: https://www.wsj.com/politics/policy/teen-deepfake-ai-nudes-bill-ted-cruz-amy-klobuchar-3106eda0 https://www.wsj.com/politics/policy/teen-deepfake-ai-nudes-b... [1]: https://www.fcc.gov/consumers/guides/deep-fake-audio-and-video-links-make-robocalls-and-scam-texts-harder-spot https://www.fcc.gov/consumers/guides/deep-fake-audio-and-vid... [2]: https://connortumbleson.com/2022/09/19/someone-is-pretending-to-be-me/ https://connortumbleson.com/2022/09/19/someone-is-pretending... [3]: https://it.ucsf.edu/aug-2023-impersonation-attacks-target-github-developers https://it.ucsf.edu/aug-2023-impersonation-attacks-target-gi... [4]: https://creativecommons.org/2023/02/17/fair-use-training-generative-ai/ https://creativecommons.org/2023/02/17/fair-use-training-gen... [5]: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
- notTooFarGone 2y agoIt's all part of the grand enshittification. Trust in anything Internet is gonna erode without any auth, anonymous internet will slowly die.
- hibikir 2y agoThis lines up well with the success rates I have seen from expert phishers. When I worked at a certain well known company with strong security, a demon called Karla would succeed at spearphishing a bit over 50% of the security team. AI now means much less skilled people can be as good as she was. Karla as a Service. We are doomed.
- richdougherty 2y ago"The cost-effective nature of AI makes it highly plausible we're moving towards an agent vs agent future." Sounds right. I assume we will all have AI agents triaging our emails trying to protect us. Maybe we will need AI to help us discern what is really true when we search for or consume information as well. The amount and quality of plausible but fake information is only going to increase. "However, the possibilities of jailbreaks and prompt injections pose a significant challenge to using language models to prevent phishing." Gives a hint at the arms race between attack and defense.
- throw10920 2y agoI don't think that there will necessarily be an arms race. Some security problems are deterministically solvable and don't need AI. For instance, there is a very good classical algorithm for preventing password brute-forcing - exponential backoff on failure per IP address, maybe with some additional per-account backoff as well. Combined with sane password rules (e.g. correct horse battery staple, not "you must have one character from every language in Madagascar), make password brute-forcing infeasible, and force attackers to try other approaches - which in the security world counts as success. No AI needed.
- ninkendo 2y agoWhat defines a successful spear phishing? Is it just clicking a link? My process when I see a sketchy email is to hover over the links to see the domain. Phishing links are obvious to anyone who understands how URLs and DNS works. But working for a typical enterprise, all links are “helpfully” rewritten to some dumbass phishing detection service, so I can no longer do this. At my current company I got what I assumed was a phishing email, I hovered over the links, saw they were pointing to some dipshit outlook phishing detection domain, and decided “what the hell, may as well click… may as well see if this phishing detection flags it” [0]… … and it turns out it was not only not legit, but it was an internal phishing test email to see whether I’d “fall for” a phishing link. Note that the test didn’t check if I’d, say, enter my credentials into a fraudulent website. It considered me to have failed if I merely clicked a link. A link to our internal phishing detection service because of course I’m not trusted to see the actual link itself (because I’d use that to check the DNS name.) I guess the threat model is that these phishers have a zero-day browser vulnerability (worth millions on auction sites) and that I’d be instantly owned the moment I clicked an outlook phishing service link, so I failed that. Also note that this was a “spear phishing” email, so it looked like any normal internal company email (in this case to a confluence page) and had my name on it. So given that it looks nearly identical to other corporate emails, and that you can’t actually see the links (they’re all rewritten), the takeaway is that you simply cannot use email to click links, ever, in a modern company with typical infosec standards. Ever ever. Zero exceptions. - [0] My threat model doesn’t include “malware installed the moment I click a link, on an up to date browser”, because I don’t believe spear phishers have those sort of vulnerabilities available to burn, given the millions of dollars that costs.
- justinl33 2y agothis research actually demonstrates that AI will reduce the phishing threat long-term, not increase it. Yes, the 50x cost reduction is scary, but it also completely commoditizes the attack vector.
- perching_aix 2y agoI'm sorry but I'm not sure I follow. How do you mean that the commoditization of spear fishing will reduce phishing threats long term? To me that implies the exact opposite would happen?
- deleted 2y ago[deleted]
- nullc 2y agoIn the future you'll create a new email account only to receive 200,000 phishing messages per second. At that point, you'll close the account because its completely useless and thus be protected from phishing. :P
- perching_aix 2y agoSounds good :D
- justinl33 2y agoExactly
- 101008 2y agoI made a purchase yesterday from Meta (Oculus). A few minutes after payment, I received an email asking to click to confirm it was me. It came from verify@verification.metamail.com, with alert@nofraud.com cc. All red flags for phishing. I googled it because it had all the purchase information, so unless a malicious actor infiltrated Meta servers, it has to be right. And it was, after googling a bit. But why do they do such things?i would expect better from Meta.
- tomashubelbauer 2y agoI experienced the exact same thing when I bought the Flipper Zero. A "hacker device" and the email communication following the sale being made was straight out of a phishing email campaign book. I don't remember the details, it has been a while, but it was wild how sketchy the emails looked. I hope they have improved the email templates since.
- frizlab 2y agoI got way worse. I was fined for leaving an unattended baggage at the train station for a bit. The fine came through an SMS message redirecting to a domain which I had to whois to verify was owned by the train company…
- lupire 2y agoYou left your ID in the baggage?
- frizlab 2y agoI left it long enough for it to be discovered but I was around. I wasn't fined on the spot for some reasons and received the fine later.
- 0xDEAFBEAD 2y agoCouldn't an attacker put in bogus whois data? Is there anything which validates that the information from whois is actually accurate?
- 015a 2y ago"Look, humans will adapt to the ever-increasing and accelerating nightmares we invent. They always have before. Technology isn't inherently evil, its how it is used that can be evil, its not our fault that we make it so accessible and cheap for evil people to use. No, we can't build safeguards, the efficient market hypothesis leaves no room for that."
- imiric 2y agoMostly accurate, except I would change the last sentence to: "We take safety very seriously. Look how much safer our SOTA model is based on our completely made up metrics. We will also delay releasing these models to the public until we ensure they're safe for everyone, or just until we need to bump up our valuation, whichever comes first."
- bennythomsson 2y agoHow did they generate these? If I try with ChatGPT then it refuses, citing a possible violation of their content policy. Even when I tell it that this is for me personally, it knows who I am, and that it's just for a test -- which obviously I could be just pretending, but again, it knows who I am but still refuses.
- qwerty2343242 2y agoYou can host open source llm offline.
- ben_w 2y agoThey team specifically "use AI agents built from GPT-4o and Claude 3.5 Sonnet". The question here is "how did they manage to do so" not "what else can do it with less effort". As those two are run by companies actively trying to prevent their tools being used nefariously, this is also what it looks like to announce they found an unpatched bug in an LLM's alignment. (Something LessWrong, where this was published, would care about much more than Hacker News).
- a1j9o94 2y agoIf you're using ChatGPT directly as opposed to the API, the system prompts could be driving it. Also, in section 3.6 of the paper, they talk about just switching fishing email, to email helps. Or said differently, tell it that it's for a marketing email, and it will gladly write personalized outreach
- atemerev 2y agoThere are many uncensored open-weights models available that you can run locally.
- bennythomsson 2y agoI'm aware what I can do. I was wondering how they did it.
- deleted 2y ago[deleted]
- serviceberry 2y agoWhile I broadly agree with the concerns about using LLMs for "commoditized", large-scale phishing, isn't the study a bit lacking? Specifically, "click through" is a pretty poor metric for success. If I receive a unique / targeted phishing email, I sure will check it out to understand what's going on and what they're after. That doesn't necessarily mean I'm falling for the actual scam.
- dwood_dev 2y agoI hate the InfoSec generated phishing tests. They all pass DKIM, SPF, etc. Some of them are very convincing. I got dinged for clicking on a convincing one that I was curious about and was 50/50 on it being legit (login from a different IP). After that, I added an auto delete rule for all the emails that have headers for our phish testing as a service provider.
- lupire 2y agoDid you report phishing before you clicked? If not, you deserve to be dinged.
- troupe 2y agoI think the idea is that it is a numbers game. If you have a way to inexpensively generate a much higher click-through rate than doing it manually, your success rate will go up with a lower investment.
- Hilift 2y agoThe average SMB company has people that act very differently with their personal email due to they need to protect their checking account that has $400 in it. But have no such measurement or reluctance with work email. They are "clickers". There are also "repeat clickers", "serial clickers", and "frequent clickers". The only thing this study is doing is automating a small part of the profiling and preparation. There was yet another study titled "Why Employees (Still) Click on Phishing Links" by NIH. (2020) https://pmc.ncbi.nlm.nih.gov/articles/PMC7005690/ https://pmc.ncbi.nlm.nih.gov/articles/PMC7005690/ Given the pathology, clicking is the visible and obvious symptom.
- Retr0id 2y agoIt's worth noting that "success" here is getting the target to click a link, and not (for example) handing over personal information or credentials.
- webdevladder 2y agoI believe I was the target of employment-flavored spear phishing a few months ago. Could have been a researcher like the OP. - 3 new email chains from different sources in a couple weeks, all similar inquiries to see if I was interested in work (I wasn't at the time, and I receive these very rarely) - escalating specificity, all referencing my online presence, the third of which I was thinking about a month later because it hit my interests squarely - only the third acknowledged my polite declining - for the third, a month after, the email and website were offline - the inquiries were quite restrained, having no links, and only asking if I was interested, and followed up tersely with an open door to my declining I have no idea what's authentic online anymore, and I think it's dangerous to operate your online life with the belief that you can discern malicious written communications with any certainty, without very strong signals like known domains. Even realtime video content is going to be a problem eventually. I suppose we'll continue to see VPN sponsorships prop up a disproportionate share of the creator economy. In other news Google routed my mom to a misleading passport renewal service. She didn't know to look for .gov. Oh well.
- nostradumbasp 2y agoImagine if models were trained for this purpose using OS-INT and reinforcement learning instead of repurposing a general model and using generic prompts from a somewhat safe guarded LLM? That's where we're headed. Bad actors paying for DDoS attacks is more or less mainstream these days. Meanwhile the success rate for phishing attacks is incredibly high and the damage is often immense. Wonder what the price for AI targeted phishing attacks would be? Automated voice impersonation attempts at social engineering, smishing, e-mails pretending to be customers, partners, etc. I bet it could be very lucrative. I could imagine a motivated high-schooler pulling off each of those sorts of "services" in a country with lax enough laws. Couple those with traditional and modern attack vectors and wow it could be really interesting.
- mtrovo 2y agoI’ve always figured those guardrails wouldn’t really hold up, but hearing that AI-based phishing can be 50 times more cost-effective than manual attacks is a serious wake-up call. We might have to rethink everything from spam filtering to overall threat detection to step up our AI defenses game.
- ttul 2y agoOrganizations should be spear phishing their employees using LLMs to test defenses and identify gaps in security awareness training and processes.
- tylerchilds 2y agothey keep saying participants, but am i missing where the targets opted into participation?
- alisonatwork 2y agoIt's probably more a reflection on me than the authors, but one thing that stood out for me in this paper is that there is a spelling mistake in the conclusion ("spar phishing"), which immediately made it come across as poorly-reviewed and got me wondering if there are other mistakes that are outside of my expertise to identify.
- xzjis 2y agoTo my understanding, papers on arXiv are NOT reviewed, which is the primary purpose of the platform. The idea, as I see it, is to allow papers to be published more quickly, at no cost, and freely accessible to everyone online. I don't think this is a bad thing, because even with peer-reviewed papers, there are cases where they can be completely fabricated but still get published. You shouldn't rely on a single paper alone and should do further research if you have doubts about its content.
- ddmf 2y agoWe had an email come in from a pension combining processor, the url they gave so that you could add information about someone's pension was similar to: employer.git.pension-details.vercell.app Why do these companies make this stuff so hard!?