4 ms·

It's way too early to make firm predictions here, but if you're not already in the field it's helpful to know there's been 20 years of effort at automating "pen

by tptacek 10mo ago

It's way too early to make firm predictions here, but if you're not already in the field it's helpful to know there's been 20 years of effort at automating "pen-testing", and the specific subset of testing this project focused on (network pentesting --- as opposed to app pentesting, which targets specifically identified network applications) is already essentially fully automated. I would expect over the medium term agent platforms to trounce un-augmented human testing teams in basically all the "routinized" pentesting tasks --- network, web, mobile, source code reviews. There are too many aspects of the work that are just perfect fits for agent loops.

jonahx 10mo ago
So the stuff that agents would excel at is essentially just the "checklist" part of the job? Check A, B, C, possibly using tools X, Y, Z, possibly multi-step checks but everything still well-defined. Whereas finding novel exploits would still be the domain of human experts?
- tptacek 10mo ago
  I'm bullish on novel exploits too but I'm much less confident in the prediction. I don't think you can do two network pentests and not immediately reach the conclusion that the need for humans to do significant chunks of that work at all is essentially a failure of automation. With more specificity: I would not be at all surprised if the "industry standard" netpen was 90%+ agent-mediated by the end of this year. But I also think that within the next 2-3 years, that will be true of web application testing as well, which is in a sense a limited (but important and widespread) instance of "novel vulnerability" discovery.
- suriya-ganesh 10mo ago
  With exploits, you'll have to go through the rote stuff of checklisting over and over, until you see aberrations across those checklist and connect the dots. If that part of the job is automated away. I wonder how the talent and skill for finding those exploits will evolve.
- socketcluster 10mo ago
  They suck at collecting the bounty money because they can't legally own a bank account.
- cookiengineer 10mo ago
  Well, agents can't discover bypass attacks because they don't have memory. That was what DNCs [1] (Differentiable Neural Computers) tried to accomplish. Correlating scan metrics with analytics is btw a great task for DNCs and what they are good at due to how their (not so precise) memory works. Not so much though at understanding branch logic and their consequences. However, I currently believe that forensic investigations will change post LLMs, because they're very good at translating arbitrary bytecode, assembly, netasm, intel asm etc syntax to example code (in any language). It doesn't have to be 100% correct in those translations, that's why LLMs can be really helpful for the discovery phase after an incident. Check out the ghidra MCP server which is insane to see real-time [2] [1] https://github.com/JoergFranke/ADNC https://github.com/JoergFranke/ADNC [2] https://github.com/LaurieWired/GhidraMCP https://github.com/LaurieWired/GhidraMCP
  - KurSix 10mo ago
    The lack of memory issue is already being solved architecturally, and ARTEMIS is a prime example. Instead of relying on the model's context window (which is "leaky"), they use structured state passed between iterations. It's not a DNC per se, but it is a functional equivalent of long-term memory. The agent remembers it tried an SQL injection an hour ago not because it's in the context, but because it's logged in its knowledge base. This allows for chaining exploits, which used to be the exclusive domain of humans
  - tptacek 10mo ago
    Can you be more specific about the kind of "bypass attack" you think an agent can't find? Like, provide a schematic example?
    - cookiengineer 10mo ago
      SSL Heartbleed is a good example. Or pretty much any vulnerability that needs understanding of how memset or malloc works, or anything where you have to use leaky functions to create a specific offset because that's where the return (eip) in assembly is, so that you can modify/exploit that jmp or cmp call. These kind of things are very hard for LLMs because they tend to forget way too much important information about both the code (in the branching sense) and the program (in the memory sense). I can't provide a schematic for this, but it's pretty common in binary exploitation CTF events, and kind of mandatory knowledge about exploit development. I listed some nice CTFs we did with our group in case you wanna know more about these things [1]. I think in regards to LLMs and this bypass/sidechannel attacks topic I'd refer to the Fusion CTF [2] specifically, because it covers a lot of examples. [1] https://cookie.engineer/about/writeups.html https://cookie.engineer/about/writeups.html [2] https://exploit.education/fusion/ https://exploit.education/fusion/
observationist 10mo ago
It's not way too early, imo. This is the academic nerds proof of concept for a school research project, it's not the "group of elite hackers get together and work out a world-class production ready system". Agent platforms have similar modes of failure, whether it's creative writing, coding, web design, hacking, or any other sort of project scaffolding. A lot of recent research has dealt with resolving the underlying gaps in architectures and training processes, and they've had great success. I fully expect frontier labs to have generalized methodologizing capabilities withing the first half of the year, and by the end of the year, the Pro/Max/Heavy variants of the chatbots will have the capabilities baked in fully. Instead of having Codex or Artemis or Claude Code, you can just ask the model to think and plan your project, whatever the domain, and get professional class results, as if an expert human was orchestrating the project. All sorts of complex visual tool use like PCB design and building plans and 3d modeling have similar process abstractions, and the decomposition and specialized task executions are very similar in principle to the generalized skills I mentioned. I think '26 is going to be exciting as hell.
Sytten 10mo ago
The app automated pentest scanners find the bottom 10-20% of vulns, no real pentester would consider them great. Agents might get us to 40%-50% range, what they are really good at is finding "signals" that the human should investigate.
- tptacek 10mo ago
  I agree with you about scanners (we banned them at Matasano), but not about the ceiling for agents. Having written agent loops for somewhat similar "surface and contextualize hypotheses from large volumes of telemetry" problems, and, of course, having delivered hundreds of application pentests: I think 80-90% of all the findings in a web pentest report, and functionally all of the findings in a netpen report, are within 12-18 months reach of agent developers.
  - nullcathedral 10mo ago
    I'd say I agree with you there for the low-hanging fruit. The deep research (there's an image filter here but we can bypass it by knowing some obscure corner of the SVG spec) is where they still fall over and need hand holding by pointing them at the browser rendering stack, specs, etc
    - hrimfaxi 10mo ago
      Until those obscure corner cases are fed into the next training round.
      - viraptor 10mo ago
        It doesn't even need to be trained. Just feed parts of the spec. I found some interesting implementation edge cases just by submitting the source and pdf spec of a chip to Claude. Not even in a fancy way.
  - EE84M3i 10mo ago
    Would be curious to hear your hypothesis on what's the remaining 10-20% that might be out of reach? Business logic bugs?

by tptacek 10mo ago

jonahx 10mo ago

So the stuff that agents would excel at is essentially just the "checklist" part of the job? Check A, B, C, possibly using tools X, Y, Z, possibly multi-step checks but everything still well-defined. Whereas finding novel exploits would still be the domain of human experts?

tptacek 10mo ago
I'm bullish on novel exploits too but I'm much less confident in the prediction. I don't think you can do two network pentests and not immediately reach the conclusion that the need for humans to do significant chunks of that work at all is essentially a failure of automation. With more specificity: I would not be at all surprised if the "industry standard" netpen was 90%+ agent-mediated by the end of this year. But I also think that within the next 2-3 years, that will be true of web application testing as well, which is in a sense a limited (but important and widespread) instance of "novel vulnerability" discovery.
suriya-ganesh 10mo ago
With exploits, you'll have to go through the rote stuff of checklisting over and over, until you see aberrations across those checklist and connect the dots. If that part of the job is automated away. I wonder how the talent and skill for finding those exploits will evolve.
socketcluster 10mo ago
They suck at collecting the bounty money because they can't legally own a bank account.
cookiengineer 10mo ago
Well, agents can't discover bypass attacks because they don't have memory. That was what DNCs [1] (Differentiable Neural Computers) tried to accomplish. Correlating scan metrics with analytics is btw a great task for DNCs and what they are good at due to how their (not so precise) memory works. Not so much though at understanding branch logic and their consequences. However, I currently believe that forensic investigations will change post LLMs, because they're very good at translating arbitrary bytecode, assembly, netasm, intel asm etc syntax to example code (in any language). It doesn't have to be 100% correct in those translations, that's why LLMs can be really helpful for the discovery phase after an incident. Check out the ghidra MCP server which is insane to see real-time [2] [1] https://github.com/JoergFranke/ADNC https://github.com/JoergFranke/ADNC [2] https://github.com/LaurieWired/GhidraMCP https://github.com/LaurieWired/GhidraMCP
- KurSix 10mo ago
  The lack of memory issue is already being solved architecturally, and ARTEMIS is a prime example. Instead of relying on the model's context window (which is "leaky"), they use structured state passed between iterations. It's not a DNC per se, but it is a functional equivalent of long-term memory. The agent remembers it tried an SQL injection an hour ago not because it's in the context, but because it's logged in its knowledge base. This allows for chaining exploits, which used to be the exclusive domain of humans
- tptacek 10mo ago
  Can you be more specific about the kind of "bypass attack" you think an agent can't find? Like, provide a schematic example?
  - cookiengineer 10mo ago
    SSL Heartbleed is a good example. Or pretty much any vulnerability that needs understanding of how memset or malloc works, or anything where you have to use leaky functions to create a specific offset because that's where the return (eip) in assembly is, so that you can modify/exploit that jmp or cmp call. These kind of things are very hard for LLMs because they tend to forget way too much important information about both the code (in the branching sense) and the program (in the memory sense). I can't provide a schematic for this, but it's pretty common in binary exploitation CTF events, and kind of mandatory knowledge about exploit development. I listed some nice CTFs we did with our group in case you wanna know more about these things [1]. I think in regards to LLMs and this bypass/sidechannel attacks topic I'd refer to the Fusion CTF [2] specifically, because it covers a lot of examples. [1] https://cookie.engineer/about/writeups.html https://cookie.engineer/about/writeups.html [2] https://exploit.education/fusion/ https://exploit.education/fusion/

observationist 10mo ago

It's not way too early, imo. This is the academic nerds proof of concept for a school research project, it's not the "group of elite hackers get together and work out a world-class production ready system". Agent platforms have similar modes of failure, whether it's creative writing, coding, web design, hacking, or any other sort of project scaffolding. A lot of recent research has dealt with resolving the underlying gaps in architectures and training processes, and they've had great success. I fully expect frontier labs to have generalized methodologizing capabilities withing the first half of the year, and by the end of the year, the Pro/Max/Heavy variants of the chatbots will have the capabilities baked in fully. Instead of having Codex or Artemis or Claude Code, you can just ask the model to think and plan your project, whatever the domain, and get professional class results, as if an expert human was orchestrating the project. All sorts of complex visual tool use like PCB design and building plans and 3d modeling have similar process abstractions, and the decomposition and specialized task executions are very similar in principle to the generalized skills I mentioned. I think '26 is going to be exciting as hell.

Sytten 10mo ago

The app automated pentest scanners find the bottom 10-20% of vulns, no real pentester would consider them great. Agents might get us to 40%-50% range, what they are really good at is finding "signals" that the human should investigate.

tptacek 10mo ago
I agree with you about scanners (we banned them at Matasano), but not about the ceiling for agents. Having written agent loops for somewhat similar "surface and contextualize hypotheses from large volumes of telemetry" problems, and, of course, having delivered hundreds of application pentests: I think 80-90% of all the findings in a web pentest report, and functionally all of the findings in a netpen report, are within 12-18 months reach of agent developers.
- nullcathedral 10mo ago
  I'd say I agree with you there for the low-hanging fruit. The deep research (there's an image filter here but we can bypass it by knowing some obscure corner of the SVG spec) is where they still fall over and need hand holding by pointing them at the browser rendering stack, specs, etc
  - hrimfaxi 10mo ago
    Until those obscure corner cases are fed into the next training round.
    - viraptor 10mo ago
      It doesn't even need to be trained. Just feed parts of the spec. I found some interesting implementation edge cases just by submitting the source and pdf spec of a chip to Claude. Not even in a fancy way.
- EE84M3i 10mo ago
  Would be curious to hear your hypothesis on what's the remaining 10-20% that might be out of reach? Business logic bugs?