5 ms·

I’m super interested since it seems like you have given everything a lot of thought and effort but I am not sure I understand it. When I’m thinking of sandboxe

I’m super interested since it seems like you have given everything a lot of thought and effort but I am not sure I understand it. When I’m thinking of sandboxes, I’m thinking of isolated execution environments. What does forking sandboxes bring me? What do your sandboxes in general bring me? Please take this in the best possible way: I’m missing a use case example that’s not abstract and/or small. What’s the end goal here(

benswerd 7mo ago
So isolation is correct. Forking a sandbox gives you multiple exact duplicates of isolated environments. When your coding agent has 10 ideas for what to do, to evaluate them correctly it needs to be able to evaluate them in isolation. If you're building a website testing agent and halfway down a website, with a form half filled out a session ongoing, etc and it realizes it wants to test 2 things in isolation, forking is the only way. We also envision this powering the next generation of devcycles "AI Agent, go try these 10 things and tell me which works best". AI forks the environment 10 times, gets 10 exact copies, does the thing in each of them, evaluates it, then takes the best option.
- indigodaddy 7mo ago
  Yep I can see this especially when the agent is spinning up test servers/smokes and you don't want those conflicting. How do we reconcile all the potential different git hashes though, upstream I guess etc (this might be an easy answer and I'm not super proficient with git so forgive)
  - benswerd 7mo ago
    So we recommend branch per fork, merge what you like. You have to change the branch on each fork individually currently and thats unlikely to change in the short term due to the complexity of git internals, but its not that hard to do yourself `git checkout -b fork-{whateverDiscriminator}`
    - chrisweekly 7mo ago
      Have you considered git worktree?
      - benswerd 7mo ago
        Great for simple things, but git worktrees don't work when you have to fork processes like postgres/complex apps.
        ghm2199 7mo ago
        For postgres there are pg containers, we use them in pytest fixtures for 1000's of unit-tests running concurrently. I imagine you could run them for integration test purposes too. What kind of testing would you run with these that can't be run with pg containers or not covered by conventional testing? I'll say this is still quite useful win for browser control usecases and also for debugging their crashes.
- mememememememo 7mo ago
  The other way might be testing VMs vs agent VMs but that would be slower as to "fork" it would need to run the test again to that point. But wouldn't need agent context. The forking you provided adds a lot more speed.
  - benswerd 7mo ago
    That + its not always simple to replicate state. A QA agent in the future could run for hours to trigger an edge case that if all actions to get there were theoretically taken again it wouldn't happen. That can happen via race conditions, edge states, external service bugs.
- vasco 7mo ago
  > and it realizes it wants to test 2 things in isolation, forking is the only way Why would forking be the only way, when humans don't work like that? You can easily try one thing, undo, try the second thing. Your way is a faster way potentially, but also uses more compute.
  - benswerd 7mo ago
    This assumes you can retain the same state after an operation. > "I wonder if this is slow because we have 100k database rows" > DELETE FROM TABLE; > "Woah its way faster now" > But was is the 100k rows or was it a specific row Thats a great place where drilling bugs and recreating exact issues can be really problem, and testing the issues themselves can be destructive to the environment leading to the need for snapshots and fork.
    - vasco 7mo ago
      Again, that is a problem of approach, not of compute. Compute just makes that faster, it doesn't make it possible. It's like you saying the only way to do something is with threads. It's good for some use cases, bad for others, and makes most faster, but it doesn't unlock much
- stingraycharles 7mo ago
  You should focus much more on this aspect, this makes so much more sense but it’s a very specific, narrow use case: multiple solution spaces must be explored in parallel, and then reconciled. I can also see this being more of a framework / library that integrates into existing LLM frameworks than a SaaS; I wouldn’t switch my whole application to a different framework / runtime just for this.
  - benswerd 7mo ago
    This is a good note. We've never been great at explaining what we're doing and plan to do a lot more work on making it accessible/make sense.
wsve 7mo ago
Agreed, the thing I'd be most interested in is the isolated execution environment you mentioned. Agents running autopilot are powerful. Agents running unsupervised on a machine with developer permissions and certificates where anything could influence the agent to act on an attacker's behalf is terrifying
- benswerd 7mo ago
  I recommend running the agent harness outside of the computer. The mental model I like to use is the computer is a tool the agent is using, and anything in the computer is untrusted.
  - croes 7mo ago
    The problem is the agent, which should be treated untrusted. The computer isn’t the problem
    - benswerd 7mo ago
      Kind of. The chat logs of the agent are trustworthly, as should any telemetry you have on it or coming out of the VM. Its behavior should be treated as probabilistic and therefore untrustworthly.
      - lll-o-lll 7mo ago
        It’s untrustworthy because its context can be poisoned and then the agent is capable of harm to the extent of whatever the “computer” you give it is capable of. The mitigation is to keep what it can do to “just the things I want it to do” (e.g. branch protection and the like, whitelisted domains/paths). And to keep all the credentials off its box and inject them inline as needed via a proxy/gateway. I mean, that’s already something you can do for humans also.
  - jeremyjh 7mo ago
    I would recommend not giving an agent the full run of any computing environment. Do handle fine grained internet access controls and credential injection like OpenShell does?

benswerd 7mo ago

So isolation is correct. Forking a sandbox gives you multiple exact duplicates of isolated environments. When your coding agent has 10 ideas for what to do, to evaluate them correctly it needs to be able to evaluate them in isolation. If you're building a website testing agent and halfway down a website, with a form half filled out a session ongoing, etc and it realizes it wants to test 2 things in isolation, forking is the only way. We also envision this powering the next generation of devcycles "AI Agent, go try these 10 things and tell me which works best". AI forks the environment 10 times, gets 10 exact copies, does the thing in each of them, evaluates it, then takes the best option.

indigodaddy 7mo ago
Yep I can see this especially when the agent is spinning up test servers/smokes and you don't want those conflicting. How do we reconcile all the potential different git hashes though, upstream I guess etc (this might be an easy answer and I'm not super proficient with git so forgive)
- benswerd 7mo ago
  So we recommend branch per fork, merge what you like. You have to change the branch on each fork individually currently and thats unlikely to change in the short term due to the complexity of git internals, but its not that hard to do yourself `git checkout -b fork-{whateverDiscriminator}`
  - chrisweekly 7mo ago
    Have you considered git worktree?
    - benswerd 7mo ago
      Great for simple things, but git worktrees don't work when you have to fork processes like postgres/complex apps.
      - ghm2199 7mo ago
        For postgres there are pg containers, we use them in pytest fixtures for 1000's of unit-tests running concurrently. I imagine you could run them for integration test purposes too. What kind of testing would you run with these that can't be run with pg containers or not covered by conventional testing? I'll say this is still quite useful win for browser control usecases and also for debugging their crashes.
mememememememo 7mo ago
The other way might be testing VMs vs agent VMs but that would be slower as to "fork" it would need to run the test again to that point. But wouldn't need agent context. The forking you provided adds a lot more speed.
- benswerd 7mo ago
  That + its not always simple to replicate state. A QA agent in the future could run for hours to trigger an edge case that if all actions to get there were theoretically taken again it wouldn't happen. That can happen via race conditions, edge states, external service bugs.
vasco 7mo ago
> and it realizes it wants to test 2 things in isolation, forking is the only way Why would forking be the only way, when humans don't work like that? You can easily try one thing, undo, try the second thing. Your way is a faster way potentially, but also uses more compute.
- benswerd 7mo ago
  This assumes you can retain the same state after an operation. > "I wonder if this is slow because we have 100k database rows" > DELETE FROM TABLE; > "Woah its way faster now" > But was is the 100k rows or was it a specific row Thats a great place where drilling bugs and recreating exact issues can be really problem, and testing the issues themselves can be destructive to the environment leading to the need for snapshots and fork.
  - vasco 7mo ago
    Again, that is a problem of approach, not of compute. Compute just makes that faster, it doesn't make it possible. It's like you saying the only way to do something is with threads. It's good for some use cases, bad for others, and makes most faster, but it doesn't unlock much
stingraycharles 7mo ago
You should focus much more on this aspect, this makes so much more sense but it’s a very specific, narrow use case: multiple solution spaces must be explored in parallel, and then reconciled. I can also see this being more of a framework / library that integrates into existing LLM frameworks than a SaaS; I wouldn’t switch my whole application to a different framework / runtime just for this.
- benswerd 7mo ago
  This is a good note. We've never been great at explaining what we're doing and plan to do a lot more work on making it accessible/make sense.

wsve 7mo ago

Agreed, the thing I'd be most interested in is the isolated execution environment you mentioned. Agents running autopilot are powerful. Agents running unsupervised on a machine with developer permissions and certificates where anything could influence the agent to act on an attacker's behalf is terrifying

benswerd 7mo ago
I recommend running the agent harness outside of the computer. The mental model I like to use is the computer is a tool the agent is using, and anything in the computer is untrusted.
- croes 7mo ago
  The problem is the agent, which should be treated untrusted. The computer isn’t the problem
  - benswerd 7mo ago
    Kind of. The chat logs of the agent are trustworthly, as should any telemetry you have on it or coming out of the VM. Its behavior should be treated as probabilistic and therefore untrustworthly.
    - lll-o-lll 7mo ago
      It’s untrustworthy because its context can be poisoned and then the agent is capable of harm to the extent of whatever the “computer” you give it is capable of. The mitigation is to keep what it can do to “just the things I want it to do” (e.g. branch protection and the like, whitelisted domains/paths). And to keep all the credentials off its box and inject them inline as needed via a proxy/gateway. I mean, that’s already something you can do for humans also.
- jeremyjh 7mo ago
  I would recommend not giving an agent the full run of any computing environment. Do handle fine grained internet access controls and credential injection like OpenShell does?