7 ms·

So let me get this straight. You’re writing tens of thousands of lines of code that will presumably go into a public GitHub repository and/or be served from som

So let me get this straight. You’re writing tens of thousands of lines of code that will presumably go into a public GitHub repository and/or be served from some location. Even if it only runs locally on your own machine, at some point you’ll presumably give that code network access. And that code is being developed (without much review) by an agent that, in our threat model, has been fully subverted by prompt injection? Sandboxing the agent hardly seems like a sufficient defense here.

simonw 1y ago
What is your worst case scenario from this?
- noitpmeder 1y ago
  Bank accounts drained, ransomware installed, ...
- deadbabe 1y ago
  Silently setup a child pornographer exchange server and run it on your machine for years without you ever noticing until you are caught and imprisoned.
- deleted 1y ago
  [deleted]
tptacek 1y ago
Where did "without much review" come from? I don't see that in the deck.
- enraged_camel 1y ago
  Yeah. Personally I haven't found a workflow that relies heavily on detailed design specs, red/green TDD followed by code review. And that's fine because that's how I did my work before AI anyway, both at the individual level and at the team level. So really, this is no different than reviewing someone else's PR, aside from the (greatly increased) turnaround time and volume.
  - deleted 1y ago
    [deleted]
  - tyre 1y ago
    I’ve found it helpful to have a model write a detailed architecture and implementation proposal, which I then review and iterate on. From there it splits out each phase into three parts: implementation, code review, and iteration. After each part, I do a code review and iteration. If asked, the proposal is broken down into small, logical chunks so code review is pretty quick. It can only stray so far off track. I treat it like a strong mid-level engineer who is learning to ship iteratively.
    - theshrike79 1y ago
      I play Claude and Codex against each other Codex is pretty good at finding complex bugs in the code, but Claude is better at getting stuff working
  - Leynos 1y ago
    That's pretty much how I use Codex.
- matthewdgreen 1y ago
  He wrote 14,000 lines of code in several days. How much review is going on there?
  - simonw 1y ago
    Oh hang on, I think I've spotted a point of confusion here. All three of the projects I described in this talk have effectively zero risk in terms of containing harmful unreviewed code. DeepSeek-OCR on the Spark? I ran that one in a Docker container, saved some notes on the process and then literally threw away the container once it had finished. The Pyodide in Node.js one I did actually review, because its code I execute on a machine that isn't disposable. The initial research ran in a disposable remote container though (Claude Code for web). The Perl in WebAssembly one? That runs in a browser sandbox. There's effectively nothing bad that can happen there, that's why I like WebAssembly so much. I am a whole lot more cautious in reviewing code that has real stakes attached to it.
    - matthewdgreen 1y ago
      Understood. I read the article as “here is how to do YOLO coding safely”, and part of the “safely” idea was to sandbox the coding agent. I’m just pointing out that this, by itself, seems insufficient to prevent ugly exfiltration, it just makes exfiltration take an extra step. I’m also not sure that human code review scales to this much code, nor that it can contain that kind of exfiltration if the instructions specify some kind of obfuscation. Obviously your recommendation to sandbox network access is one of several you make (the most effective one being “don’t let the agent ever touch sensitive data”), so I’m not saying the combined set of protections won’t work well. I’m also not saying that your projects specifically have any risk, just that they illustrate how much code you can end up with very quickly — making human review a fool’s errand. ETA: if you do think human review can prevent secret exfiltration, I’d love to turn that into some kind of competition. Think of it as the obfuscated C contest with a scarier twist.
      - tptacek 1y ago
        Is it your claim that LLMs will produce subtly obfuscated secret exfiltrations?
daxfohl 1y ago
That's kind of tangential though. The article is more about using sandboxes to allow `--dangerously-skip-permissions` mode. If you're not looking at the generated code, you're correct, sandboxing doesn't help, but neither does permissioning, so it's not directly relevant to the main point.
- hypercube33 1y ago
  I've noted a few tools seem to have this now if you dig into their settings. vscode has yolo mode and cursor seemed to have something as well. wild
- matthewdgreen 1y ago
  My point is that if the threat model is “agent got sensitive information and wants to exfiltrate it”, looking at the code isn’t going to save you. You’re already dead. This was the threat outlined in TFA.

simonw 1y ago

What is your worst case scenario from this?

noitpmeder 1y ago
Bank accounts drained, ransomware installed, ...
deadbabe 1y ago
Silently setup a child pornographer exchange server and run it on your machine for years without you ever noticing until you are caught and imprisoned.
deleted 1y ago
[deleted]

tptacek 1y ago

Where did "without much review" come from? I don't see that in the deck.

enraged_camel 1y ago
Yeah. Personally I haven't found a workflow that relies heavily on detailed design specs, red/green TDD followed by code review. And that's fine because that's how I did my work before AI anyway, both at the individual level and at the team level. So really, this is no different than reviewing someone else's PR, aside from the (greatly increased) turnaround time and volume.
- deleted 1y ago
  [deleted]
- tyre 1y ago
  I’ve found it helpful to have a model write a detailed architecture and implementation proposal, which I then review and iterate on. From there it splits out each phase into three parts: implementation, code review, and iteration. After each part, I do a code review and iteration. If asked, the proposal is broken down into small, logical chunks so code review is pretty quick. It can only stray so far off track. I treat it like a strong mid-level engineer who is learning to ship iteratively.
  - theshrike79 1y ago
    I play Claude and Codex against each other Codex is pretty good at finding complex bugs in the code, but Claude is better at getting stuff working
- Leynos 1y ago
  That's pretty much how I use Codex.
matthewdgreen 1y ago
He wrote 14,000 lines of code in several days. How much review is going on there?
- simonw 1y ago
  Oh hang on, I think I've spotted a point of confusion here. All three of the projects I described in this talk have effectively zero risk in terms of containing harmful unreviewed code. DeepSeek-OCR on the Spark? I ran that one in a Docker container, saved some notes on the process and then literally threw away the container once it had finished. The Pyodide in Node.js one I did actually review, because its code I execute on a machine that isn't disposable. The initial research ran in a disposable remote container though (Claude Code for web). The Perl in WebAssembly one? That runs in a browser sandbox. There's effectively nothing bad that can happen there, that's why I like WebAssembly so much. I am a whole lot more cautious in reviewing code that has real stakes attached to it.
  - matthewdgreen 1y ago
    Understood. I read the article as “here is how to do YOLO coding safely”, and part of the “safely” idea was to sandbox the coding agent. I’m just pointing out that this, by itself, seems insufficient to prevent ugly exfiltration, it just makes exfiltration take an extra step. I’m also not sure that human code review scales to this much code, nor that it can contain that kind of exfiltration if the instructions specify some kind of obfuscation. Obviously your recommendation to sandbox network access is one of several you make (the most effective one being “don’t let the agent ever touch sensitive data”), so I’m not saying the combined set of protections won’t work well. I’m also not saying that your projects specifically have any risk, just that they illustrate how much code you can end up with very quickly — making human review a fool’s errand. ETA: if you do think human review can prevent secret exfiltration, I’d love to turn that into some kind of competition. Think of it as the obfuscated C contest with a scarier twist.
    - tptacek 1y ago
      Is it your claim that LLMs will produce subtly obfuscated secret exfiltrations?

daxfohl 1y ago

That's kind of tangential though. The article is more about using sandboxes to allow `--dangerously-skip-permissions` mode. If you're not looking at the generated code, you're correct, sandboxing doesn't help, but neither does permissioning, so it's not directly relevant to the main point.

hypercube33 1y ago
I've noted a few tools seem to have this now if you dig into their settings. vscode has yolo mode and cursor seemed to have something as well. wild
matthewdgreen 1y ago
My point is that if the threat model is “agent got sensitive information and wants to exfiltrate it”, looking at the code isn’t going to save you. You’re already dead. This was the threat outlined in TFA.