8 ms·
Teams of LLM Agents Can Exploit Zero-Day Vulnerabilities
- randomllm 2y agoTell me who will pay a pentester $50 an hour to look for zero days in these targets?
- beardedwizard 2y agois this working because agents are just another way of chunking context across models to keep them from drifting off topic?
- spdustin 2y agoAgents "work" because (in a well-designed agentic system) their context isn't polluted by tokens that aren't meaningful to the work they're doing. Their attention mechanisms are less likely to drift around with each new token if the subject matter largely stays the same.
- beardedwizard 2y agook so exactly what I said then. im sure its just me (and not directed at your reply), but I find 'agent' to be a very obnoxious term that obfuscates what is really going on. It's not a magical being working on your behalf with agency, it's that there are inherent limitations to attention, and overcoming them usually involves having multiple models cooperate on the inputs and outputs. the number of agents is not necessarily related to the number of distinct jobs you need done, after all a single job could still perform best with multiple "agents" due to limitations in attention.
- mewpmewp2 2y agoMultiple models cooperating doesn't sound much better though. In the end it could be a same model or even a same single instance where it just gets triggered with different prompts in sequence. Response to first prompt decides 10 new tasks to be done in sequence with different prompts, running them then doing a final prompt with the results of those.
- beardedwizard 2y agoThat's a fair point, multiple models is a potential implementation detail for parallelization - I do think it's fair to assume you are "restarting" the model or clearing context by some means between runs otherwise I don't think it achieves the goal of groomed attention.
- mewpmewp2 2y agoThat gets really philosophical here. What is a single run actually? Because each token is generated one by one, with all the previous tokens as input. If we remove one token from the context in the past does it make it a new "run"? What if we have a summarization memory system, where in order to keep the context size small after certain length it will start to summarize/compress the beginning until the context size is good. Then the whole input and context is constantly changing/evolving. You could have 100s of different randomly selected instances generating a new token one by one to the (last input + token generated by another instance) as input.
- beardedwizard 2y agoI think it comes down to the goal you are trying to achieve: maximize use of available attention while minimizing drift. The experiment seems clear, and the outcome being tested is the resulting generation.
- simonw 2y agoThe term "agents" is often a distraction in my opinion. It's usually a fancy way of saying a prompt with function calls - but since the definition is vague it leads to different people holding different mental models from each other. As far as I can tell, this team took descriptions of vulnerabilities in the form of CVEs and fed those into a system consisting of various GPT-4 Turbo prompts-with -functions configured for things like executing Playwright commands or running code in a terminal. They arranged this such that there was a "planner" prompt which could then delegate to prompts specializing in XSS or SQLi. These were the "agents". I imagine they won't be sharing their code, which is a shame because it would be interesting to see the details of how they got this to work.
- cess11 2y agoWe'll soon have FOSS adapters on top of LLM:s or similar software capable of managing and to some degree extend botnets. It's likely something like this will leak from a state actor during the coming years too.
- squigz 2y agoWhy would I need an LLM for running a botnet?
- Sharlin 2y agoWhat decade are you from, the 2010s? Today you need an LLM, preferably two, to do anything. Just like half a decade ago you needed a blockchain to do anything.
- michaelbuckbee 2y agoObviously, the answer is you don't. My personal concerns are more that we are going to see the botnets get smarter and smarter and more automated. Just imagine if every script kiddie running a "dumb" botnet suddenly got 2x more effective.
- szundi 2y ago
- nomilk 2y ago> However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities) Only a matter of time until LLMs are brilliant enough to find sophisticated real-world vulnerabilities. Perhaps not every novelty, but certainly vulnerabilities that resemble any seen in the past. The good news is in this arms race devs will be armed with similar tools, and the solution will be a few lines of CI code to run our own vulnerability-seeking LLM and prevent merging when (not if) vulnerabilities are detected.
- DelightOne 2y agoThe open source maintainer gonna pay this?
- nomilk 2y ago> pay For a few API calls?
- DelightOne 2y agoThe proposed approach used multiple LLM chains. Then you don't know how much context you need. I imagine it can get expensive. Per Commit.
- nomilk 2y agoOpenAI's latest (GPT-4o) model is US$5.00 / 1M tokens [1] For large repos it might get expensive, but many will have orgs footing the bill. To satiate curiosity, I made a fresh rails app and it has 9141023 characters, let's /5 to estimate 'tokens' (wild guess), so to scan an entire rails app, that's about $10. Not nothing, but not back-breakingly expensive. Scanning could be reserved for non-trivial PRs and important applications where vulnerabilities are especially likely (e.g. perhaps not static sites, demos, or apps not in prod or a live environment). Scanning only new or edited files, along with those they interact with (where sensible patterns could emerge over time) could decrease the total volume of files needing scans, thereby reducing costs. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
- brigadier132 2y ago> LLM agents have become increasingly sophisticated, especially in the realm of cybersecurity. Researchers have shown that LLM agents can exploit real-world vulnerabilities when given a description of the vulnerability and toy capture-the-flag problems. However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities). Does this perform any better than the million bots that are out there exploiting vulnerabilities without AI?
- TeMPOraL 2y ago> Does this perform any better than the million bots that are out there exploiting vulnerabilities without AI? Right now? Probably not. But I wouldn't dismiss it because of it, as the "million bots that are out there" are hand-coded to exploit specific vulnerabilities; GPT-4 has just been fed with half the internet and asked to take a crack at the problem. That's the difference between a special-purpose and general-purpose system. It's not unreasonable to believe LLMs will get better at this, and eventually might be able to identify and exploit novel vulnerabilities on the fly.
- deleted 2y ago[deleted]
- brigadier132 2y ago> But I wouldn't dismiss it because of it, as the "million bots that are out there" are hand-coded to exploit specific vulnerabilities I would absolutely dismiss it if it's literally just an LLM sitting on top of bots and because there is an LLM involved now it's suddenly an agent. What is the LLM doing here that makes it more potent than existing techniques? > However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities). This quote is the tell This article seems like a prime example of the perverse incentives that exist in academia.
- szundi 2y agoThey could be faster as LLMs will create 1000 exploits for every vulnerability in 5 minutes and then those that work are massively used in 6 minutes. No time to install security updates in time. Of course we could create 1000 security fixes in 5 minutes, then create 10k exploits with LLMs, run functional tests and attack all 1k with the 10k and hopefully you have a tried fix in 30 minutes.
- sva_ 2y agoImagine a self-replicating code that is able to find zero-days and spread through the internet, finding niches of Turing-completeness offering compute keeping itself spreading.
- tdullien 2y agoI think there are more than enough people with limited domain understanding imagining that AI can do all sorts of things that it won't be able to do any time soon, if ever. There's enough Yudkowskis fabulating the impossible and screaming about it. The bugs in the paper are largely low complexity issues that have been auto-exploitable by domain-specific programs in the past. I really wish everything Ai would get a much more sober and less hyperbolic treatment.
- broknbottle 2y agoHow else do you propose that we pump the stock? If it’s not blockchain, NFTs, AI, GenAI, then we’ll need something. Stock won’t pump itself.
- 1oooqooq 2y agodepressing part is the current pump beats buying coca cola.
- beardedwizard 2y agothis. all these articles written about LLMs exploiting zero days completely miss the point. No researcher uses it this way, they use it to explore the semantic meaning of the code to discover and confirm the vulnerabilities. It is a more easily used SAST.
- mattgreenrocks 2y agoWhen established SMEs in VR or other domains start getting good results with LLMs, I’m happy to take a look. Until then, I remain suspicious of too-excited opinions on this topic because it is socially cheap and advantageous to grab onto hype waves, especially ones of this magnitude. It is also easy to underestimate the knowledge of SMEs if you don’t have the requisite domain knowledge yourself, and believe that LLMs get those details right.
- EthanHeilman 2y agoI've asked Claude-opus to find vulnerabilities in well known code bases and explain what the bug is and what code change should be used to fix it. So far it has identified issues that are technically correct but aren't actually an issue within the larger context of the how the code is used. That said, I'm pretty sure I could find vulnerabilities by using Claude-opus as first pass filter to find problem areas. ChatGPT broke a cryptographic protocol I wrote[0]. I convinced myself ChatGPT was wrong about the attack until a human cryptographer pointed the same attack out [1]. "The protocol you described appears to be a variant of the Schnorr signature scheme, with a zero-knowledge proof of knowledge of the signature. However, this specific variant does not provide security against forgery attacks by the prover. The reason for this is that the prover can choose a random value r and compute w = r^e mod N, without actually computing the signature sig = h(m)^d mod N. Then, the prover can simply choose zksig = w and publish (w, zksig, m) as the proof. Since zksig = w^d mod N = r^(ed) mod N, it satisfies the verification equation zksig^e mod N = h(m)*w mod N, even though it is not a valid signature for the message m." [0]: https://chatgpt.com/share/f5526fd5-7ebb-4b15-a2bf-703ac88de455 https://chatgpt.com/share/f5526fd5-7ebb-4b15-a2bf-703ac88de4... [1]: https://crypto.stackexchange.com/questions/105704/nizk-proof-of-knowledge-of-a-standard-rsa-signature-on-a-message-without-signer https://crypto.stackexchange.com/questions/105704/nizk-proof...
- qup 2y agoHave you tried putting a past, now patched, bug through [LLM] on purpose, as a test?
- faitswulff 2y agoIf it’s in the past on a public repo there’s a good chance it’s in the training data.
- btdmaster 2y ago> all vulnerabilities were past the GPT-4 cutoff date at the time of experimentation I thought that was proven inaccurate at some point? Like the model has certain information mislabelled by date, so has more newer information than the stated cutoff would suggest.
- mewpmewp2 2y agoMaybe through chat, but supposedly through API the models should be frozen, they are versioned by a date. Otherwise they would be much more difficult to use in automated manners, if they might be changing slightly throughout the time.
- woodruffw 2y agoI think LLMs have a lot of promise for automated vulnerability research. However, to pick at two small things: 1. The vulnerability classes chosen (XSS, CSRF, SQLi, etc.) have well known, basic "shapes" that don't vary as much in the wild as other classes, e.g. memory corruption or logic bugs. The latter also have a common subset of shapes, but frequently require long sequences of dependent operations to reach an exploitable state. It would be interesting to see more research tackle these vulnerability classes! 2. The standard for "proof of ignorance" in the field is cutoff dates: it's taken for granted that, because GPT-4's training cutoff predates any of the CVEs, they must not be included in the training set. This might be appropriate in some cases, but it's harder to confidently assert for publicly disclosed vulnerabilities: a public CVE may be subject to extensive semi-public documentation and conversation before being made "public" in the form of a CVE, and it'd be interesting to know whether any of that has been included. Edit: To make the point in (2) more clear: it's somewhat common for a CVE to come from a public issue tracker, where a user reports a bug or piece of unexpected behavior that ends up being exploitable. That issue tracker doesn't get labeled with the CVE itself (from the LLM's cutoff perspective at least), but it contains much of the information needed to derive the exploit. In such a case, it's hard to establish that the LLM has successfully produced a novel exploit rather than just regurgitating what it was trained on.
- bongodongobob 2y agoI don't think that's how LLMs train is it? One instance of a bug in an issue tracker shouldn't be enough for the LLM to "remember". For example, if I were to post my password is ($;3+$($7262 there's basically a 0% likelyhood that it will ever infer "bongodongobobs password on hackernews is ($;3+$($7262". It's not a database of bits of knowledge.
- woodruffw 2y agoTo my understanding: one of the biggest outstanding research areas around LLMs is that we don't actually understanding all that well how much they "remember" vs. infer. This is at the centerpiece of the ongoing lawsuits from newspapers against OpenAI: ChatGPT didn't just summarize articles, but appears to be capable of reproducing them verbatim[1]. (But even separately from this: the LLM doesn't need to store verbatim details from the pre-CVE-labeled vulnerability to poison the experiment here. Being trained on it at all poisons the experiment, since it's no longer an 0day.) [1]: https://www.nytimes.com/2024/04/30/business/media/newspapers-sued-microsoft-openai.html https://www.nytimes.com/2024/04/30/business/media/newspapers...
- msephton 2y agoI wonder if they could also be used to optimise code? So that iOS can become faster and less memory hungry?
- deleted 2y ago[deleted]