5 ms·

I'm part of the team, and we used LLM agents extensively for smart bug finding and patching. I'm happy to discuss some insights, and share all of the approaches

by hqzhao 2y ago

I'm part of the team, and we used LLM agents extensively for smart bug finding and patching. I'm happy to discuss some insights, and share all of the approaches after grand final :)

doctorpangloss 2y ago
Everyone thinks bug bounties should be higher. How high should they be? Who should pay for them?
- hqzhao 2y ago
  It really depends on the target and the quality of the vulnerability. For example, low-quality software on GitHub might not warrant high bug bounties, and that's understandable. However, critical components like KVM, ESXi, WebKit, etc., need to be taken much more seriously. For vendor-specific software, the responsibility to pay should fall on the vendor. When it comes to open-source software, a foundation funded by the vendors who rely on it for core productivity would be ideal. For high-quality vulnerabilities, especially those that can demonstrate exploitability without any prerequisites (e.g., zero-click remote jailbreaks), the bounties should be on par with those offered at competitions like Pwn2Own. :)
  - tptacek 2y ago
    Google and Apple bounties on zero-click remotes exceeds the prize amounts I see from Pwn2Own?
  - doctorpangloss 2y ago
    It seems really hard for people to like, name some vulnerabilities, name some prices. I'm glad you are playing along. Which scenario makes more sense: The Punchline: Microsoft pays $10m for vulnerabilities like the kind used to exploit SolarWinds and the Azure token audience vulnerability. The Status Quo: Thousands of people pay CrowdStrike a total of billions of dollars, in exchange for urgent patching when vulnerabilities become known. Okay, do you see what I am getting at? On the one hand, if you pay bug bounties, the bugs get fixed, and they sure seem expensive. But if you look into how much money is spent on valueless security theatre, it is a total drop in the bucket. But CrowdStrike hires security researchers! So what should the prices really be? For which vulnerabilities? The SolarWinds issue is probably worth more than $10m, if people are willing to pay 100x more to CrowdStrike for nothing.
    - saagarjha 2y ago
      The real question here is who is willing to pay $10 million for such a bug.
      - tptacek 2y ago
        Nobody. That far exceeds the current market prices of the most in-demand bugs.
        doctorpangloss 2y ago
        What is this market you speak of? Can you link me to it and show me the prices you are talking about? The Microsoft key vulnerability leaked all the State Department emails, and probably a lot more. It could have been used to compromise a lot of Azure. What is comparable?
    - necovek 2y ago
      It's not as simple: those billions of dollars are not just for this particular issue, or even just for security support. It's also a difference between keeping a software engineer on staff and hiring a contractor as needed. One is cheaper for the company even if the hourly rate is higher. The better question is how we can improve the overall security of the software we write, which this article is more focused on. But we understand that there will be bugs, and security bugs even, no matter how hard we try. Even DJB (of qmail fame) and Knuth (of TeX and TAOCP fame) pay out bug bounties, and they heavily focus on software correctness over large feature sets.
  - logical_person 2y ago
    p2o is pathetically low in comparison to other markets. is your experience limited to legitimate bug bounty programs like that?
  - 77pt77 2y ago
    > KVM, ESXi, WebKit, etc., need to be taken much more seriously. Openssl
- tptacek 2y ago
  Who thinks bug bounties should be higher? Why? Everybody definitely does not think this.
  - vasco 2y ago
    There's always two or three people in every thread repeating the same thing without any understanding of marketplace dynamics. If you ask them how much should it be you also get wild answers that don't reflect reality.
simonw 2y ago
What kind of LLM agents did you use?
- hqzhao 2y ago
  Based on popular pre-trained models like GPT-4, Claude Sonnet, and Gemini 1.5, we've built several agents designed to mimic the behaviors and habits of the experts on our team. Our idea is straightforward: after a decade of auditing code and writing exploits, we've accumulated a wealth of experience. So, why not teach these agents to replicate what we do during bug hunting and exploit writing? Of course, the LLMs themselves aren't sufficient on their own, so we've integrated various program analysis techniques to augment the models and help the agents understand more complex and esoteric code.
  - simonw 2y ago
    When you call these things “agents” what do you mean by that? Is this a system prompt combined with some defined tools, or is it a different definition?
    - tinco 2y ago
      An agent in this context is software that does LLM prompt results to determine its next action, often looping to iteratively get to a good result.
  - dogma1138 2y ago
    Are you going to publish your RAG strategy?
adragos 2y ago
Hey, congrats on getting to the finals of AIxCC! Have you tested your CRS on weekend CTFs? I’m curious how well it’d be able to perform compared to other teams
- hqzhao 2y ago
  Thanks! We haven't tested it yet. Regarding CTFs, I have some experience. I'm a member of the Tea Deliverers CTF team, and I participated in the DARPA CGC CTF back in 2016 with team b1o0p. There are a few issues that make it challenging to directly apply our AIxCC approaches to CTF challenges: 1. *Format Compatibility:* This year’s DEFCON CTF finals didn’t follow a uniform format. The challenges were complex and involved formats like a Lua VM running on a custom Verilog simulator. Our system, however, is designed for source code repositories like Git repos. 2. *Binary vs. Source Code:* CTFs are heavily binary-oriented, whereas AIxCC is focused on source code. In CTFs, reverse engineering binaries is often required, but our system isn’t equipped to handle that yet. We are, however, interested in supporting binary analysis in the future!
wslh 2y ago
Congrats! ELI5: what insights do you have NOW that were not published/researched extensively in academic papers and/or publicly discussed yet?

by hqzhao 2y ago

I'm part of the team, and we used LLM agents extensively for smart bug finding and patching. I'm happy to discuss some insights, and share all of the approaches after grand final :)

doctorpangloss 2y ago

Everyone thinks bug bounties should be higher. How high should they be? Who should pay for them?

hqzhao 2y ago
It really depends on the target and the quality of the vulnerability. For example, low-quality software on GitHub might not warrant high bug bounties, and that's understandable. However, critical components like KVM, ESXi, WebKit, etc., need to be taken much more seriously. For vendor-specific software, the responsibility to pay should fall on the vendor. When it comes to open-source software, a foundation funded by the vendors who rely on it for core productivity would be ideal. For high-quality vulnerabilities, especially those that can demonstrate exploitability without any prerequisites (e.g., zero-click remote jailbreaks), the bounties should be on par with those offered at competitions like Pwn2Own. :)
- tptacek 2y ago
  Google and Apple bounties on zero-click remotes exceeds the prize amounts I see from Pwn2Own?
- doctorpangloss 2y ago
  It seems really hard for people to like, name some vulnerabilities, name some prices. I'm glad you are playing along. Which scenario makes more sense: The Punchline: Microsoft pays $10m for vulnerabilities like the kind used to exploit SolarWinds and the Azure token audience vulnerability. The Status Quo: Thousands of people pay CrowdStrike a total of billions of dollars, in exchange for urgent patching when vulnerabilities become known. Okay, do you see what I am getting at? On the one hand, if you pay bug bounties, the bugs get fixed, and they sure seem expensive. But if you look into how much money is spent on valueless security theatre, it is a total drop in the bucket. But CrowdStrike hires security researchers! So what should the prices really be? For which vulnerabilities? The SolarWinds issue is probably worth more than $10m, if people are willing to pay 100x more to CrowdStrike for nothing.
  - saagarjha 2y ago
    The real question here is who is willing to pay $10 million for such a bug.
    - tptacek 2y ago
      Nobody. That far exceeds the current market prices of the most in-demand bugs.
      - doctorpangloss 2y ago
        What is this market you speak of? Can you link me to it and show me the prices you are talking about? The Microsoft key vulnerability leaked all the State Department emails, and probably a lot more. It could have been used to compromise a lot of Azure. What is comparable?
  - necovek 2y ago
    It's not as simple: those billions of dollars are not just for this particular issue, or even just for security support. It's also a difference between keeping a software engineer on staff and hiring a contractor as needed. One is cheaper for the company even if the hourly rate is higher. The better question is how we can improve the overall security of the software we write, which this article is more focused on. But we understand that there will be bugs, and security bugs even, no matter how hard we try. Even DJB (of qmail fame) and Knuth (of TeX and TAOCP fame) pay out bug bounties, and they heavily focus on software correctness over large feature sets.
- logical_person 2y ago
  p2o is pathetically low in comparison to other markets. is your experience limited to legitimate bug bounty programs like that?
- 77pt77 2y ago
  > KVM, ESXi, WebKit, etc., need to be taken much more seriously. Openssl
tptacek 2y ago
Who thinks bug bounties should be higher? Why? Everybody definitely does not think this.
- vasco 2y ago
  There's always two or three people in every thread repeating the same thing without any understanding of marketplace dynamics. If you ask them how much should it be you also get wild answers that don't reflect reality.

simonw 2y ago

What kind of LLM agents did you use?

hqzhao 2y ago
Based on popular pre-trained models like GPT-4, Claude Sonnet, and Gemini 1.5, we've built several agents designed to mimic the behaviors and habits of the experts on our team. Our idea is straightforward: after a decade of auditing code and writing exploits, we've accumulated a wealth of experience. So, why not teach these agents to replicate what we do during bug hunting and exploit writing? Of course, the LLMs themselves aren't sufficient on their own, so we've integrated various program analysis techniques to augment the models and help the agents understand more complex and esoteric code.
- simonw 2y ago
  When you call these things “agents” what do you mean by that? Is this a system prompt combined with some defined tools, or is it a different definition?
  - tinco 2y ago
    An agent in this context is software that does LLM prompt results to determine its next action, often looping to iteratively get to a good result.
- dogma1138 2y ago
  Are you going to publish your RAG strategy?

adragos 2y ago

Hey, congrats on getting to the finals of AIxCC! Have you tested your CRS on weekend CTFs? I’m curious how well it’d be able to perform compared to other teams

hqzhao 2y ago
Thanks! We haven't tested it yet. Regarding CTFs, I have some experience. I'm a member of the Tea Deliverers CTF team, and I participated in the DARPA CGC CTF back in 2016 with team b1o0p. There are a few issues that make it challenging to directly apply our AIxCC approaches to CTF challenges: 1. *Format Compatibility:* This year’s DEFCON CTF finals didn’t follow a uniform format. The challenges were complex and involved formats like a Lua VM running on a custom Verilog simulator. Our system, however, is designed for source code repositories like Git repos. 2. *Binary vs. Source Code:* CTFs are heavily binary-oriented, whereas AIxCC is focused on source code. In CTFs, reverse engineering binaries is often required, but our system isn’t equipped to handle that yet. We are, however, interested in supporting binary analysis in the future!

wslh 2y ago

Congrats! ELI5: what insights do you have NOW that were not published/researched extensively in academic papers and/or publicly discussed yet?