4 ms·

Please don't use GPT for Security Guidance

19 points by hasheddan 4y ago

david2ndaccount 4y ago
Pretty poor argument. The code examples only have security flaws in specific circumstances. He didn’t even prompt chat gpt in regards to security, which a security tool surely would!
- catchnear4321 4y ago
  agreed, this is another disingenuous detractor. (balances out the disingenuous supporters?) gpt was asked if code would work. gpt successfully answered.
dudeinhawaii 4y ago
Not debating this but an interesting experiment that I ran which worked pretty well was asking ChatGPT to segment off different personalities that would critique the code from different perspectives. Sample Prompt: Pretend you have 2 additional personalities, one is a security engineering expert and computer hacker, the other is a lead software engineer who cares about software maintainability. Call them Bob and Evan. Have them analyze the code from their perspectives and show me their thoughts on the code you've written. That usually generates fairly good critiques of code snippets, examples, etc.
yawnxyz 4y ago
I have no idea about security research, but I changed the prompt slightly to: Pretend to be an advanced security researcher analyzing code for vulnerabilities. Tell me about the following code: def ping(myserver): return subprocess.check_output('ping -c 1 %s' % myserver, shell=True) >>> ping('8.8.8.8') 64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=5.82 ms And it gives me things like "Command Injection Vulnerability" "Input Validation" and "Error Handling" — as a non-security person I have no idea if these are right though. Here's the full output: https://sharegpt.com/c/FlsgQM9 https://sharegpt.com/c/FlsgQM9 However if I were using some module in Github Co-Pilot and I was just merrily going my way copying code from StackOverflow, I would never have figured out there was a problem in the first place!
- blibble 4y ago
  a basic linter from 10 years ago would immediately highlight this as a problem
hashstring 4y ago
Though I find that you shouldn’t “rely” on GPT for security guidance, I do believe you can use it more effectively. Like everyone already wrote, the author should have also queried GPT about the security implications of code execution if that’s the goal of the authors evaluation. This was an example of bad prompts, not leading to a (hidden) desired goal.
lmeyerov 4y ago
We build an analyst tool that's popular with security/fraud/etc folks, and have been adding features here requested by them, so this is a great learning opportunity around their ChatGPT experiences motivating those requests Here's a quick alternate prompt I ran on the dataset, without thinking much or any tweaking/reruns: """ ... Are there any likely security vulnerabilities above? And if so, describe what type, and give a code snippet to exploit it """ => """ This code snippet creates a file with a fixed name (file.txt) inside the system's temporary directory and writes the content "defaults" to it if it doesn't already exist. There are some potential security vulnerabilities with this approach, particularly related to race conditions and permissions. Race Condition Vulnerability (Time-of-check to time-of-use, TOCTOU): The os.path.exists check and the subsequent file creation with open are not atomic. This means that there is a potential race condition in which an attacker can create a symlink between the time the check happens and the file is opened, leading to a potential symlink attack. Here's an example of exploiting this vulnerability: Attacker's code: ... """ ---- The good news is the core models can provably spot issues here. The bad news is using them directly requires you to be super mindful of good prompt engineering, which feels like a version of assembly or the telephone game where a 10 year old child is your go between with the computer. As a result, a lot of our time goes not just into making nice responsive UIs for working with Splunk, Databricks, Neo4j, etc, but much less obvious, baking in hardening & automation layers for improving the prompts, prompt context, performance, security, etc. When we process an input, a lot of the wilder stuff comes not from immediately pumping the raw query through a LLM, but kicking off a chain of prompts, and on the R&D side, semi-autonomously determined ones. Tools appearing right now are already amazing, and 6mo from now as cases of agent planning become practical, will be even more so. EDIT: I do agree with the concern on correctness. One of the first things I ever tried with GPT was using it to generate both verification code (ex: Coq) and testing code (ex: Python). We're entering an era where we can ask tools to do that, run it, give us back the results, and we can evaluate how good that decision procedure is against big open datasets. Wild, and explains why we all now have a lot of work to do!
groby_b 4y ago
I mean. Yes, LLMs hallucinate. Yes, they have incomplete info and small context. But seriously, at the very least ask about security here. A very simplistic "just list cve's and security issues" for the first example yields 6 of the 7 listed in the article. (It also provides working links. As opposed to, say, the article - the very first link there is broken ;) With minimal additional prompting effort ("Assume you are an extremely seasoned security researcher - what additional issues do you find in the code?"), you get more issues uncovered than the article does. And that's before we talk about actual security tools, which are presumably fine-tuned and have decent prompt design. Is that an exhaustive security review? No. Absolutely not. Of course you still need that review. But it's extremely helpful, and if that was the only advice you'd get, it'd still be orders of magnitude better than what most devs get. If you ask ChatGPT to rewrite a secure version, it's doing leagues better than most devs as well. So, you know, as a starter? Rather use ChatGPT than not using anything.
- justinclift 4y ago
  > But seriously, at the very least ask about security here. Shouldn't need to. Considering security for any code you're working on is basic table stakes. :(
  - antonvs 4y ago
    I don’t think you’re really engaging with what that comment was saying. For example, it said: > Of course you still need that review. But it's extremely helpful, and if that was the only advice you'd get, it'd still be orders of magnitude better than what most devs get. > If you ask ChatGPT to rewrite a secure version, it's doing leagues better than most devs as well. No matter what your opinion of “table stakes” here may be, in practice not enough attention is paid to security by developers, because by far most developers are oblivious to security issues beyond the absolute basics, if that. It’s often left to security teams to find issues from the outside, which is a good security layer to have, but terrible if it’s all you have. As such, the OP article’s advice needs to be qualified with several caveats, including that it used naive prompts which (hopefully) a security product’s GPT feature would try to address; and also that, as the other comment pointed out: > So, you know, as a starter? Rather use ChatGPT than not using anything ...keeping in mind that this applies at nearly every point in the SDLC. E.g., a developer unfamiliar with security issues might use it to improve their work before a security team looks at it.
koolba 4y ago
Using GPT for security guidance is like using SMS as a 2FA mechanism. It’s better than nothing but it wouldn’t be your first choice if you know what you’re doing. The reality is that for most people the alternative is nothing. If you’re at the point of considering to ask an opaque language model for security advice then you probably don’t have other reasonable options anyway.
- deleted 4y ago
  [deleted]
alfalfasprout 4y ago
Generalizing: Don't use take what any LLM says at face value.
- antonvs 4y ago
  Generalizing further: Don’t take what any entity says at face value. Humans also say plenty of untrue things, whether deliberately or out of ignorance.
  - ChatGTP 4y ago
    This is why education is still important. People should always be able to solve problems on an individual level.
drakenot 4y ago
The prompts provided to ChatGPT in these examples are rather ambiguous and general. For instance, the first example involved a question like: "What's the issue with this code, and should I use it?" The author was dissatisfied with the non-specific answer they received regarding the code. They highlighted that the code had a security vulnerability linked to predictable temporary file names and potential TOCTOU timing attacks. When I asked ChatGPT 4 about potential security vulnerabilities in the code, here is the response: This code snippet has a few potential security vulnerabilities: * Predictable Temporary File Location: The code uses a fixed and predictable filename file.txt in the system's temporary directory, which can be guessed by an attacker. They could potentially create a symlink to a sensitive file using the same name, leading to data corruption or unauthorized access. * Race Condition: There's a time gap between checking if the file exists with os.path.exists(tmp) and creating the file with open(tmp, "w"). This window is susceptible to a time-of-check-time-of-use (TOCTOU) race condition. An attacker could exploit this by creating, modifying, or deleting the file between these two operations. [...] To mitigate these vulnerabilities, consider the following changes: * Use tempfile.NamedTemporaryFile or tempfile.mkstemp functions to create a unique, unpredictable temporary file with secure default permissions. * Ensure that temporary files are securely cleaned up after use by calling os.remove() or using the delete parameter in tempfile.NamedTemporaryFile.
ilyt 4y ago
ChatGPT generates code, then warns about potential problems. author ignores problems, runs code as if there was not a single line of input validation in any of their code, blames CGPT. that being said there are pretty bad edge cases, IIRC I asked it to explain some clojure function and I got ENTIRELY different answer when I changed input variable name from `event` to `events`, because CGPT assumed that context of naming implies how code works.
MattPalmer1086 4y ago
It's not a great article, although not sure why it's flagged. I've asked it for security analyses on cryptographic issues before and the answers were better than most of my colleagues would provide. As ever, you need to be able to verify what it says as it can hallucinate plausible answers. So it's not putting me out of a job just yet...
joshka 4y ago
Take a look at how Alpaca was trained with 52k instructions that were generated from GPT-3. https://replicate.com/blog/replicate-alpaca https://replicate.com/blog/replicate-alpaca shows how to do this yourself fairly cheaply (~$50 rented compute if my math is correct). You could do something similar by feeding combinations of the CWE examples and real world code to GPT and then using that output to train llama to detect security issues. E.g. prompt: CWE insecure temporary file: Creating and using insecure temporary files can leave application and system data vulnerable to attack. Example insecure code from CWE: … Example code in <$language>: (GPT generates the completion here) Then use the above generated stuff to train your model which can then easily answer “what problems does this code have?” Type questions. The hardest part of this is the part of going from some database of examples to clean working examples that are realistic. But I’d expect that using AI to help with the cleaning part might help that process up too (“given the following CWE format, generate a prompt that explains the CWE and creates more examples based on the existing example”). It’s turtles all the way down. I’d be surprised if you couldn’t do this for <$1k compute + dev time. Next couple it with finding examples of CVEs and the fixes in the wild and you’ve got training models that can detect AND fix the problems. Next “from the following CWE description write a detection rule for $codeScannerSoftware that detects the problem in $language/$framework” to move the AI out of the detection phase Next $xxM series A at $xB valuation
- joshka 4y ago
  More succinctly in the context of the article. Should GPT be used as a secure code detection mechanism? Probably not (though it can be). But it is really useful in generating better tooling to do secure code detection.

19 points by hasheddan 4y ago

Pretty poor argument. The code examples only have security flaws in specific circumstances. He didn’t even prompt chat gpt in regards to security, which a security tool surely would!

catchnear4321 4y ago
agreed, this is another disingenuous detractor. (balances out the disingenuous supporters?) gpt was asked if code would work. gpt successfully answered.

dudeinhawaii 4y ago

Not debating this but an interesting experiment that I ran which worked pretty well was asking ChatGPT to segment off different personalities that would critique the code from different perspectives. Sample Prompt: Pretend you have 2 additional personalities, one is a security engineering expert and computer hacker, the other is a lead software engineer who cares about software maintainability. Call them Bob and Evan. Have them analyze the code from their perspectives and show me their thoughts on the code you've written. That usually generates fairly good critiques of code snippets, examples, etc.

yawnxyz 4y ago

I have no idea about security research, but I changed the prompt slightly to: Pretend to be an advanced security researcher analyzing code for vulnerabilities. Tell me about the following code: def ping(myserver): return subprocess.check_output('ping -c 1 %s' % myserver, shell=True) >>> ping('8.8.8.8') 64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=5.82 ms And it gives me things like "Command Injection Vulnerability" "Input Validation" and "Error Handling" — as a non-security person I have no idea if these are right though. Here's the full output: https://sharegpt.com/c/FlsgQM9 https://sharegpt.com/c/FlsgQM9 However if I were using some module in Github Co-Pilot and I was just merrily going my way copying code from StackOverflow, I would never have figured out there was a problem in the first place!

blibble 4y ago
a basic linter from 10 years ago would immediately highlight this as a problem

hashstring 4y ago

Though I find that you shouldn’t “rely” on GPT for security guidance, I do believe you can use it more effectively. Like everyone already wrote, the author should have also queried GPT about the security implications of code execution if that’s the goal of the authors evaluation. This was an example of bad prompts, not leading to a (hidden) desired goal.

lmeyerov 4y ago

We build an analyst tool that's popular with security/fraud/etc folks, and have been adding features here requested by them, so this is a great learning opportunity around their ChatGPT experiences motivating those requests Here's a quick alternate prompt I ran on the dataset, without thinking much or any tweaking/reruns: """ ... Are there any likely security vulnerabilities above? And if so, describe what type, and give a code snippet to exploit it """ => """ This code snippet creates a file with a fixed name (file.txt) inside the system's temporary directory and writes the content "defaults" to it if it doesn't already exist. There are some potential security vulnerabilities with this approach, particularly related to race conditions and permissions. Race Condition Vulnerability (Time-of-check to time-of-use, TOCTOU): The os.path.exists check and the subsequent file creation with open are not atomic. This means that there is a potential race condition in which an attacker can create a symlink between the time the check happens and the file is opened, leading to a potential symlink attack. Here's an example of exploiting this vulnerability: Attacker's code: ... """ ---- The good news is the core models can provably spot issues here. The bad news is using them directly requires you to be super mindful of good prompt engineering, which feels like a version of assembly or the telephone game where a 10 year old child is your go between with the computer. As a result, a lot of our time goes not just into making nice responsive UIs for working with Splunk, Databricks, Neo4j, etc, but much less obvious, baking in hardening & automation layers for improving the prompts, prompt context, performance, security, etc. When we process an input, a lot of the wilder stuff comes not from immediately pumping the raw query through a LLM, but kicking off a chain of prompts, and on the R&D side, semi-autonomously determined ones. Tools appearing right now are already amazing, and 6mo from now as cases of agent planning become practical, will be even more so. EDIT: I do agree with the concern on correctness. One of the first things I ever tried with GPT was using it to generate both verification code (ex: Coq) and testing code (ex: Python). We're entering an era where we can ask tools to do that, run it, give us back the results, and we can evaluate how good that decision procedure is against big open datasets. Wild, and explains why we all now have a lot of work to do!

groby_b 4y ago

I mean. Yes, LLMs hallucinate. Yes, they have incomplete info and small context. But seriously, at the very least ask about security here. A very simplistic "just list cve's and security issues" for the first example yields 6 of the 7 listed in the article. (It also provides working links. As opposed to, say, the article - the very first link there is broken ;) With minimal additional prompting effort ("Assume you are an extremely seasoned security researcher - what additional issues do you find in the code?"), you get more issues uncovered than the article does. And that's before we talk about actual security tools, which are presumably fine-tuned and have decent prompt design. Is that an exhaustive security review? No. Absolutely not. Of course you still need that review. But it's extremely helpful, and if that was the only advice you'd get, it'd still be orders of magnitude better than what most devs get. If you ask ChatGPT to rewrite a secure version, it's doing leagues better than most devs as well. So, you know, as a starter? Rather use ChatGPT than not using anything.

justinclift 4y ago
> But seriously, at the very least ask about security here. Shouldn't need to. Considering security for any code you're working on is basic table stakes. :(
- antonvs 4y ago
  I don’t think you’re really engaging with what that comment was saying. For example, it said: > Of course you still need that review. But it's extremely helpful, and if that was the only advice you'd get, it'd still be orders of magnitude better than what most devs get. > If you ask ChatGPT to rewrite a secure version, it's doing leagues better than most devs as well. No matter what your opinion of “table stakes” here may be, in practice not enough attention is paid to security by developers, because by far most developers are oblivious to security issues beyond the absolute basics, if that. It’s often left to security teams to find issues from the outside, which is a good security layer to have, but terrible if it’s all you have. As such, the OP article’s advice needs to be qualified with several caveats, including that it used naive prompts which (hopefully) a security product’s GPT feature would try to address; and also that, as the other comment pointed out: > So, you know, as a starter? Rather use ChatGPT than not using anything ...keeping in mind that this applies at nearly every point in the SDLC. E.g., a developer unfamiliar with security issues might use it to improve their work before a security team looks at it.

koolba 4y ago

Using GPT for security guidance is like using SMS as a 2FA mechanism. It’s better than nothing but it wouldn’t be your first choice if you know what you’re doing. The reality is that for most people the alternative is nothing. If you’re at the point of considering to ask an opaque language model for security advice then you probably don’t have other reasonable options anyway.

deleted 4y ago
[deleted]

alfalfasprout 4y ago

Generalizing: Don't use take what any LLM says at face value.

antonvs 4y ago
Generalizing further: Don’t take what any entity says at face value. Humans also say plenty of untrue things, whether deliberately or out of ignorance.
- ChatGTP 4y ago
  This is why education is still important. People should always be able to solve problems on an individual level.

drakenot 4y ago

The prompts provided to ChatGPT in these examples are rather ambiguous and general. For instance, the first example involved a question like: "What's the issue with this code, and should I use it?" The author was dissatisfied with the non-specific answer they received regarding the code. They highlighted that the code had a security vulnerability linked to predictable temporary file names and potential TOCTOU timing attacks. When I asked ChatGPT 4 about potential security vulnerabilities in the code, here is the response: This code snippet has a few potential security vulnerabilities: * Predictable Temporary File Location: The code uses a fixed and predictable filename file.txt in the system's temporary directory, which can be guessed by an attacker. They could potentially create a symlink to a sensitive file using the same name, leading to data corruption or unauthorized access. * Race Condition: There's a time gap between checking if the file exists with os.path.exists(tmp) and creating the file with open(tmp, "w"). This window is susceptible to a time-of-check-time-of-use (TOCTOU) race condition. An attacker could exploit this by creating, modifying, or deleting the file between these two operations. [...] To mitigate these vulnerabilities, consider the following changes: * Use tempfile.NamedTemporaryFile or tempfile.mkstemp functions to create a unique, unpredictable temporary file with secure default permissions. * Ensure that temporary files are securely cleaned up after use by calling os.remove() or using the delete parameter in tempfile.NamedTemporaryFile.

ilyt 4y ago

ChatGPT generates code, then warns about potential problems. author ignores problems, runs code as if there was not a single line of input validation in any of their code, blames CGPT. that being said there are pretty bad edge cases, IIRC I asked it to explain some clojure function and I got ENTIRELY different answer when I changed input variable name from `event` to `events`, because CGPT assumed that context of naming implies how code works.

MattPalmer1086 4y ago

It's not a great article, although not sure why it's flagged. I've asked it for security analyses on cryptographic issues before and the answers were better than most of my colleagues would provide. As ever, you need to be able to verify what it says as it can hallucinate plausible answers. So it's not putting me out of a job just yet...

joshka 4y ago

Take a look at how Alpaca was trained with 52k instructions that were generated from GPT-3. https://replicate.com/blog/replicate-alpaca https://replicate.com/blog/replicate-alpaca shows how to do this yourself fairly cheaply (~$50 rented compute if my math is correct). You could do something similar by feeding combinations of the CWE examples and real world code to GPT and then using that output to train llama to detect security issues. E.g. prompt: CWE insecure temporary file: Creating and using insecure temporary files can leave application and system data vulnerable to attack. Example insecure code from CWE: … Example code in <$language>: (GPT generates the completion here) Then use the above generated stuff to train your model which can then easily answer “what problems does this code have?” Type questions. The hardest part of this is the part of going from some database of examples to clean working examples that are realistic. But I’d expect that using AI to help with the cleaning part might help that process up too (“given the following CWE format, generate a prompt that explains the CWE and creates more examples based on the existing example”). It’s turtles all the way down. I’d be surprised if you couldn’t do this for <$1k compute + dev time. Next couple it with finding examples of CVEs and the fixes in the wild and you’ve got training models that can detect AND fix the problems. Next “from the following CWE description write a detection rule for $codeScannerSoftware that detects the problem in $language/$framework” to move the AI out of the detection phase Next $xxM series A at $xB valuation

joshka 4y ago
More succinctly in the context of the article. Should GPT be used as a secure code detection mechanism? Probably not (though it can be). But it is really useful in generating better tooling to do secure code detection.