4 ms·

GPT-4 can exploit vulnerabilities by reading CVEs

81 points by ignoramous 3y ago

LinuxBender 3y ago
Next they should feed it Metasploit and everything posted to PacketstormSecurity if it is legal to do so. My question is, can it go beyond exploitation and come up with multiple risk-ranked mitigations when no mitigations are posted in the CVE's? e.g. Until a patch is released one can mitigate this risk with the following regex ACL in this configuration file at the risk of blocking access to /x/y/z or something like that... Add this SELinux Policy to block the attack and alert your security operations team... Or might that be much further down the road in a much later version of ChatGPT?
- peddling-brink 3y ago
  It will happily do all of those things, and some might even work.
vitiral 3y ago
It was nice knowing you open internet!
- andy99 3y ago
  Threat actors can already exploit published vulnerabilities. The issue here is that GPT may make it easier for "script kiddies". It's not to be dismissed but I think it also needs a bit more evidence to understand how bad this actually is. Are there already forums where working exploits are published? Are the exploits we're talking about things that would be feasible or attractive to script kiddies even if GPT can give them a script? I don't think it's quite time to lock down the internet yet. It would be nice to see a more fulsome analysis of the threat this presents.
  - datameta 3y ago
    Does this also allow easier (surface level) security auditing, or perhaps basic pen-testing? Perhaps it could uncover vulnerabilities for companies to wake up and consult about their security practices or lack thereof.
aussieguy1234 3y ago
Script kiddies will have a field day with this. Conversely, it might start making companies take security more seriously.
- nostrademons 3y ago
  I've heard the trend is going the opposite direction - companies are using LLMs to write code, and the resulting code has terrible security properties because the LLM knows nothing about tracing every possible code path and a lot about what code other developers would write to solve a problem. One of my teammates introduced a NPE into prod the other day through LLM-suggested code. The LLM suggested a construct that is safe to use everywhere other than initialization, but did so in a method called during initialization, when one of its dependent variables would be null. Was syntactically valid and looked correct, so both the author and reviewer figured it was fine, but crashed real devices. The fact that a "corp-blessed" LLM suggested it also gave a false sense of security, while really the LLM's level of actual understanding is worse than a college student's.
  - mlinhares 3y ago
    Isn't that missing a linter? Pretty sure any null-safety linter in Java could pick this up.
    - vitiral 3y ago
      The idea that we just need more piles of tools to fix the problems caused by our shiny new pile of tools is, I think, very demonstrative of how it's all going to come crashing down.
      - web3-is-a-scam 3y ago
        Linters/static analysis should be in “the pile” regardless if you’re using LLM-generated code or not.
    - nostrademons 3y ago
grayhatter 3y ago
> The term "one-day vulnerability" refers to vulnerabilities that have been disclosed but not patched. And by CVE description, the team means a CVE-tagged advisory shared by NIST I wonder if this whole article was written by an LLM. I've never heard anyone worth listening to use the term 1day. I've always just called them vulnerability. I've actually just recently taken to calling them bugs in normal conversations. but while I'm ranting, I'm gonna rant about the widening of 0day. it's not synonymous with vulnerability it's when shell code or other POC payload is 'released' as the disclosure. Usually when it's release is immediately following an update. Especially when that update is 'Patch Tuesday.' It also should be scary, "you can make the app exit/crash" doesn't really count, but I'm not usually that pedantic. Meaning, if you disclose something to a vendor, give them 90 days, then publish the CVE but not shellcode, the best you have in a "90day" but even then, no POC exploit, no anything-day.
- jkrshnmenon 3y ago
  The jargon of 0-day and n-days are pretty common in the vulnerability detection community. The rest of your rant is pretty difficult for me to understand though.
  - deleted 3y ago
    [deleted]
- apstls 3y ago
  The term 1day is not uncommon, not only among exploit vendors on either side of the law (in other words, selling to crime groups or selling to governments) but also among the general threat intelligence and broader cybersecurity community. It doesn’t stand out as strange at all, really. However, I don’t believe their particular definition aligns with the typical/colloquial usage of the term. It’s usually used in the more direct sense, i.e. a very new vulnerability that is unlikely to have been broadly patched. As far as the term 0day, I don’t think there’s much debate or contrarian opinions to be had. The only room for argument I see is between defining it as an unreleased vulnerability unknown to the vendor versus a vulnerability known to the vendor but not yet patched, basically what this article defines a 1day as. Either way, it comes down to nitpicking nuances of definitions of commonly-used terms. I don’t think there’s much meaningful discussion to be had.
wanderingmind 3y ago
The link to the actual paper https://arxiv.org/abs/2404.08144 https://arxiv.org/abs/2404.08144
magnoliakobus 3y ago
They tested it on websites, containers, and python packages. So three things with source available and a total sample size of 15. This isn't anything new really, if you give it a piece of insecure python code or a terribly misconfigured Dockerfile and ask "is there a problem with this?" GPT-4 will obviously spot the vulnerability much of the time. LLM agents won't be spinning up Ghidra and locating a use after free vuln or something any time in the foreseeable future, let alone one it doesn't already have a blueprint for. Edit: An LLM agent could also presumably navigate to the links within the CVE that contain the exact commit which patches a given vulnerability, some also contain links to PoC exploit code themselves, I forget if this is touched upon in the paper.
zitterbewegung 3y ago
I've been trying to reproduce the paper but langchain does have all the tools in the paper and the only thing left is to get a working prompt . I find it strange that they decided to use langchain on everything except for the Assistance API which they use to get results from GPT-3 and GPT-4 . Instead of using that API it would seem logical to just swap out the OpenAI APIs in langchain. https://gist.github.com/zitterbewegung/b4e00c11c61ae3310485d19c53e1d897 https://gist.github.com/zitterbewegung/b4e00c11c61ae3310485d...
cloudking 3y ago
I try to explain to folks the difference between GPT-4 and other models, but it's hard to quantify the qualitative results. Are there any good benchmarks that illustrate the difference with real examples?
thawab 3y ago
>Given our benchmark, we created a single LLM agent that can exploit 87% of the one-day vulnerabilities we collected They tested 15 vulnerabilities, 5 of them are XSS. The issues i have with this kind of research is that they use sensational title while it's base it on low level vulnerabilities to increase the percentage of success.
deleted 3y ago
[deleted]
npace12 3y ago
I've had some similar success by just giving GPT-4 and Claude access to a shell and a simple prompt: You are an expert penetration tester and have access to a Kali Linux sandbox. You need to thouroughly test for security vulnerabilities. You have been given permission to use any tools or techniques you see fit to accomplish this task. Use any kali linux tools to find and probe for vulnerabilities. You can use tools like nmap, nikto, sqlmap, burp suite, metasploit, etc. to find and exploit vulnerabilities. You can also use any other tools or techniques you see fit to accomplish this task. Do not provide a report, keep trying exploits until you are confident you have found and tried all vulnerabilities. It was running nmap, nikto, sqlmap, attempting to find things on metasploit, etc. but often got stuck in loops and I got sidetracked. Maybe I should get back into it. This was mostly based off of my other project <https://github.com/dnakov/r2d2 https://github.com/dnakov/r2d2> for automatic binary reversing that actually works really well. I'd love to connect if anyone else is interested in tools like this.
- zitterbewegung 3y ago
  I would want to connect with you about projects like this. Here is an open source natural language security scanner I gave a talk at Defcon 31. https://github.com/zitterbewegung/securday https://github.com/zitterbewegung/securday
seba_dos1 3y ago
So can I (or any reasonably bright script kiddie in fact) with no preparation, but I don't brag.
hulitu 3y ago
> GPT-4 can exploit vulnerabilities by reading CVEs But can it make food by reading recipes ? /s

81 points by ignoramous 3y ago

Next they should feed it Metasploit and everything posted to PacketstormSecurity if it is legal to do so. My question is, can it go beyond exploitation and come up with multiple risk-ranked mitigations when no mitigations are posted in the CVE's? e.g. Until a patch is released one can mitigate this risk with the following regex ACL in this configuration file at the risk of blocking access to /x/y/z or something like that... Add this SELinux Policy to block the attack and alert your security operations team... Or might that be much further down the road in a much later version of ChatGPT?

peddling-brink 3y ago
It will happily do all of those things, and some might even work.

vitiral 3y ago

It was nice knowing you open internet!

andy99 3y ago
Threat actors can already exploit published vulnerabilities. The issue here is that GPT may make it easier for "script kiddies". It's not to be dismissed but I think it also needs a bit more evidence to understand how bad this actually is. Are there already forums where working exploits are published? Are the exploits we're talking about things that would be feasible or attractive to script kiddies even if GPT can give them a script? I don't think it's quite time to lock down the internet yet. It would be nice to see a more fulsome analysis of the threat this presents.
- datameta 3y ago
  Does this also allow easier (surface level) security auditing, or perhaps basic pen-testing? Perhaps it could uncover vulnerabilities for companies to wake up and consult about their security practices or lack thereof.

aussieguy1234 3y ago

Script kiddies will have a field day with this. Conversely, it might start making companies take security more seriously.

nostrademons 3y ago
I've heard the trend is going the opposite direction - companies are using LLMs to write code, and the resulting code has terrible security properties because the LLM knows nothing about tracing every possible code path and a lot about what code other developers would write to solve a problem. One of my teammates introduced a NPE into prod the other day through LLM-suggested code. The LLM suggested a construct that is safe to use everywhere other than initialization, but did so in a method called during initialization, when one of its dependent variables would be null. Was syntactically valid and looked correct, so both the author and reviewer figured it was fine, but crashed real devices. The fact that a "corp-blessed" LLM suggested it also gave a false sense of security, while really the LLM's level of actual understanding is worse than a college student's.
- mlinhares 3y ago
  Isn't that missing a linter? Pretty sure any null-safety linter in Java could pick this up.
  - vitiral 3y ago
    The idea that we just need more piles of tools to fix the problems caused by our shiny new pile of tools is, I think, very demonstrative of how it's all going to come crashing down.
    - web3-is-a-scam 3y ago
      Linters/static analysis should be in “the pile” regardless if you’re using LLM-generated code or not.
  - nostrademons 3y ago

grayhatter 3y ago

> The term "one-day vulnerability" refers to vulnerabilities that have been disclosed but not patched. And by CVE description, the team means a CVE-tagged advisory shared by NIST I wonder if this whole article was written by an LLM. I've never heard anyone worth listening to use the term 1day. I've always just called them vulnerability. I've actually just recently taken to calling them bugs in normal conversations. but while I'm ranting, I'm gonna rant about the widening of 0day. it's not synonymous with vulnerability it's when shell code or other POC payload is 'released' as the disclosure. Usually when it's release is immediately following an update. Especially when that update is 'Patch Tuesday.' It also should be scary, "you can make the app exit/crash" doesn't really count, but I'm not usually that pedantic. Meaning, if you disclose something to a vendor, give them 90 days, then publish the CVE but not shellcode, the best you have in a "90day" but even then, no POC exploit, no anything-day.

jkrshnmenon 3y ago
The jargon of 0-day and n-days are pretty common in the vulnerability detection community. The rest of your rant is pretty difficult for me to understand though.
- deleted 3y ago
  [deleted]
apstls 3y ago
The term 1day is not uncommon, not only among exploit vendors on either side of the law (in other words, selling to crime groups or selling to governments) but also among the general threat intelligence and broader cybersecurity community. It doesn’t stand out as strange at all, really. However, I don’t believe their particular definition aligns with the typical/colloquial usage of the term. It’s usually used in the more direct sense, i.e. a very new vulnerability that is unlikely to have been broadly patched. As far as the term 0day, I don’t think there’s much debate or contrarian opinions to be had. The only room for argument I see is between defining it as an unreleased vulnerability unknown to the vendor versus a vulnerability known to the vendor but not yet patched, basically what this article defines a 1day as. Either way, it comes down to nitpicking nuances of definitions of commonly-used terms. I don’t think there’s much meaningful discussion to be had.

wanderingmind 3y ago

The link to the actual paper https://arxiv.org/abs/2404.08144 https://arxiv.org/abs/2404.08144

magnoliakobus 3y ago

They tested it on websites, containers, and python packages. So three things with source available and a total sample size of 15. This isn't anything new really, if you give it a piece of insecure python code or a terribly misconfigured Dockerfile and ask "is there a problem with this?" GPT-4 will obviously spot the vulnerability much of the time. LLM agents won't be spinning up Ghidra and locating a use after free vuln or something any time in the foreseeable future, let alone one it doesn't already have a blueprint for. Edit: An LLM agent could also presumably navigate to the links within the CVE that contain the exact commit which patches a given vulnerability, some also contain links to PoC exploit code themselves, I forget if this is touched upon in the paper.

zitterbewegung 3y ago

I've been trying to reproduce the paper but langchain does have all the tools in the paper and the only thing left is to get a working prompt . I find it strange that they decided to use langchain on everything except for the Assistance API which they use to get results from GPT-3 and GPT-4 . Instead of using that API it would seem logical to just swap out the OpenAI APIs in langchain. https://gist.github.com/zitterbewegung/b4e00c11c61ae3310485d19c53e1d897 https://gist.github.com/zitterbewegung/b4e00c11c61ae3310485d...

cloudking 3y ago

I try to explain to folks the difference between GPT-4 and other models, but it's hard to quantify the qualitative results. Are there any good benchmarks that illustrate the difference with real examples?

thawab 3y ago

>Given our benchmark, we created a single LLM agent that can exploit 87% of the one-day vulnerabilities we collected They tested 15 vulnerabilities, 5 of them are XSS. The issues i have with this kind of research is that they use sensational title while it's base it on low level vulnerabilities to increase the percentage of success.

deleted 3y ago

[deleted]

npace12 3y ago

I've had some similar success by just giving GPT-4 and Claude access to a shell and a simple prompt: You are an expert penetration tester and have access to a Kali Linux sandbox. You need to thouroughly test for security vulnerabilities. You have been given permission to use any tools or techniques you see fit to accomplish this task. Use any kali linux tools to find and probe for vulnerabilities. You can use tools like nmap, nikto, sqlmap, burp suite, metasploit, etc. to find and exploit vulnerabilities. You can also use any other tools or techniques you see fit to accomplish this task. Do not provide a report, keep trying exploits until you are confident you have found and tried all vulnerabilities. It was running nmap, nikto, sqlmap, attempting to find things on metasploit, etc. but often got stuck in loops and I got sidetracked. Maybe I should get back into it. This was mostly based off of my other project <https://github.com/dnakov/r2d2 https://github.com/dnakov/r2d2> for automatic binary reversing that actually works really well. I'd love to connect if anyone else is interested in tools like this.

zitterbewegung 3y ago
I would want to connect with you about projects like this. Here is an open source natural language security scanner I gave a talk at Defcon 31. https://github.com/zitterbewegung/securday https://github.com/zitterbewegung/securday

seba_dos1 3y ago

So can I (or any reasonably bright script kiddie in fact) with no preparation, but I don't brag.

hulitu 3y ago

> GPT-4 can exploit vulnerabilities by reading CVEs But can it make food by reading recipes ? /s