3 ms·

Nevermind designing _systems_ that account for this, even just debugging such errors is much harder than ones you create yourself: https://www.bugsink.com/blog

Nevermind designing _systems_ that account for this, even just debugging such errors is much harder than ones you create yourself: https://www.bugsink.com/blog/copilot-induced-crash/ https://www.bugsink.com/blog/copilot-induced-crash/

fragmede 2y ago
For that case, it sounds more like having your tools commit for you after each change, as is the default for Aider, is the real winner. "git log -p" would have exposed that crazy import in minutes instead of hours. commit early, commit often.
- zahlman 2y ago
  FTA: > Note that it took me about two hours to debug this, despite the problem being freshly introduced. (Because I hadn’t committed yet, and had established that the previous commit was fine, I could have just run git diff to see what had changed). > In fact, I did run git diff and git diff --staged multiple times. But who would think to look at the import statements? The import statement is the last place you’d expect a bug to be introduced.
  - fragmede 2y ago
    git diff != git log. To expand on that, the problem with only having git diff is there's no way to go backwards halfway. You can't step backwards in time until you get to the bad commit just before the good commit, and then do a precise diff between the two. (aka git bisect) Reviewing 300 lines out of git diff and trying to find the bug somewhere in there is harder than when there are only 10.
- danenania 2y ago
  I’m working an AI coding agent[1], and all changes accumulate in a sandbox by default that is isolated from the project. Auto-commit is also enabled (by default) when you do apply the changes to your project, but I think keeping them separated until you review is better for higher stakes work and goes a long way to protect you from stray edits getting left behind. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex
  - fragmede 2y ago
    If you implement the sandbox as a git branch, then we're on the same page.
    - danenania 2y ago
      It's built on top of git, but offers better separation imho than just a separate branch. For one thing, you have to always remember to check out that branch before you start making changes with the LLM. It's easy to forget. Second, even if you're on a branch, it doesn't protect you from your own changes getting interleaved with the model's changes. You can get into a situation where you can't easily roll back and instead have to pick apart your work and the model's output. By defaulting to the sandbox, it 'just works' and you can be sure that nothing will end up in the codebase without being checked first.
      - fragmede 2y ago
        If the latest change is bad, how do you go back in your sandbox? How do you go back three steps? If you make a change outside the sandbox, how do you copy it in? How do you copy them out? How do you deinterleave the changes then? In order for this sandbox to actually be useful, you're going to end up implementing a source control mechanism. If you're going to do that, might as well just use git, even if just on the backend and commit to a branch behind the scenes that the user never sees, or by using worktree, or any other pieces of it. Take a good long think about how this sandbox will actually work in practice. Switch to the sandbox, LLM some code, save it, handwrite some code, then switch to the sandbox again, LLM some code, switch out. Try and go backwards half the LLM change. Wish you'd committed the LLM changes while you were working on the. By the time you've got a handle on it, rembering to switch git branch is the least of your troubles.
        danenania 2y ago
        This is all implemented and working, just to be clear, and is being used in production. Everything you mentioned in your comment is covered. You can also create branches within the sandbox to try different approaches, again with no risk of anything being left behind in your project until it’s ready. It does use git underneath. Here are some more details if you’re interested: https://docs.plandex.ai/core-concepts/version-control https://docs.plandex.ai/core-concepts/version-control

fragmede 2y ago

For that case, it sounds more like having your tools commit for you after each change, as is the default for Aider, is the real winner. "git log -p" would have exposed that crazy import in minutes instead of hours. commit early, commit often.

zahlman 2y ago
FTA: > Note that it took me about two hours to debug this, despite the problem being freshly introduced. (Because I hadn’t committed yet, and had established that the previous commit was fine, I could have just run git diff to see what had changed). > In fact, I did run git diff and git diff --staged multiple times. But who would think to look at the import statements? The import statement is the last place you’d expect a bug to be introduced.
- fragmede 2y ago
  git diff != git log. To expand on that, the problem with only having git diff is there's no way to go backwards halfway. You can't step backwards in time until you get to the bad commit just before the good commit, and then do a precise diff between the two. (aka git bisect) Reviewing 300 lines out of git diff and trying to find the bug somewhere in there is harder than when there are only 10.
danenania 2y ago
I’m working an AI coding agent[1], and all changes accumulate in a sandbox by default that is isolated from the project. Auto-commit is also enabled (by default) when you do apply the changes to your project, but I think keeping them separated until you review is better for higher stakes work and goes a long way to protect you from stray edits getting left behind. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex
- fragmede 2y ago
  If you implement the sandbox as a git branch, then we're on the same page.
  - danenania 2y ago
    It's built on top of git, but offers better separation imho than just a separate branch. For one thing, you have to always remember to check out that branch before you start making changes with the LLM. It's easy to forget. Second, even if you're on a branch, it doesn't protect you from your own changes getting interleaved with the model's changes. You can get into a situation where you can't easily roll back and instead have to pick apart your work and the model's output. By defaulting to the sandbox, it 'just works' and you can be sure that nothing will end up in the codebase without being checked first.
    - fragmede 2y ago
      If the latest change is bad, how do you go back in your sandbox? How do you go back three steps? If you make a change outside the sandbox, how do you copy it in? How do you copy them out? How do you deinterleave the changes then? In order for this sandbox to actually be useful, you're going to end up implementing a source control mechanism. If you're going to do that, might as well just use git, even if just on the backend and commit to a branch behind the scenes that the user never sees, or by using worktree, or any other pieces of it. Take a good long think about how this sandbox will actually work in practice. Switch to the sandbox, LLM some code, save it, handwrite some code, then switch to the sandbox again, LLM some code, switch out. Try and go backwards half the LLM change. Wish you'd committed the LLM changes while you were working on the. By the time you've got a handle on it, rembering to switch git branch is the least of your troubles.
      - danenania 2y ago
        This is all implemented and working, just to be clear, and is being used in production. Everything you mentioned in your comment is covered. You can also create branches within the sandbox to try different approaches, again with no risk of anything being left behind in your project until it’s ready. It does use git underneath. Here are some more details if you’re interested: https://docs.plandex.ai/core-concepts/version-control https://docs.plandex.ai/core-concepts/version-control