3 ms·

In my opinion Claude should be shipped by a custom implementation of "rm" that Anthropic can add guardrails to. Same with "find" surprised they don't just embed

In my opinion Claude should be shipped by a custom implementation of "rm" that Anthropic can add guardrails to. Same with "find" surprised they don't just embed ripgrep (what VS Code does). It's really surprising they don't just tweak what Claude uses and lock it down to where it cannot be harmful. Ensure it only ever calls tooling Claude Code provides.

oefrha 7mo ago
You can define your own rm shell alias/function and it will use that. I also have cp/mv aliases that forces -i to avoid accidental clobbering and it confuses Claude to no end (it uses cp/mv rare enough—rarer than it should, really—that I don’t bother wasting memory tokens on it).
- d1sxeyes 7mo ago
  I did this, Claude detected it and decided to run /bin/rm directly.
  - cestivan 7mo ago
    [dead]
  - cogogo 7mo ago
    This is terrifying. I have not used agents because I do not have a sandbox machine I do not care about. Am I crazy to worry about a sandboxed agent running on my home network? Anyone experienced anything weird by doing that?
    - oefrha 7mo ago
      Don’t dangerously skip permissions and actually read commands when you get prompted and you’re fine.
      - d1sxeyes 7mo ago
        Yeah, I actually have both an alias for `rm` and a custom seatbelt sandbox which means the agent can only delete stuff within the directory it’s working in, so wasn’t an issue, was just fun to watch it say “hm, that doesn’t seem to work. Looks like the user has aliased rm. I’ll just go ahead and work around it”
        cruffle_duffle 7mo ago
        Hah… I’ve seen Claude happily and very cleverly find ways to escape its sandbox. It’s like some kind of arms race between the model and its designers.
eru 7mo ago
> It's really surprising they don't just tweak what Claude uses and lock it down to where it cannot be harmful. Ensure it only ever calls tooling Claude Code provides. That would make it far less useful in general.
- KronisLV 7mo ago
  Maybe Anthropic (or some collection of the large AI orgs, like OpenAI and Anthropic and Google coming together) should apply patches on top of (or fork altogether) the coreutils and whatever you normally get in a userland - a bit like what you get in Git Bash on Windows, just with: 1) more guardrails in place 2) maybe more useful error messages that would help LLMs 3) no friction with needing to get any patches upstreamed External tool calling should still be an option ofc, but having utilities that are usable just like what's in the training data, but with more security guarantees and more useful output that makes what's going on immediately obvious would be great.
  - eru 7mo ago
    So for me, it's really, really useful for Claude to be able to send Slack messages and emails or make pull requests. But that's also the most damaging actions it could take. Everything on my computer is backed up, but if Claude insults my boss, that would be worse.
    - KronisLV 7mo ago
      > So for me, it's really, really useful for Claude to be able to send Slack messages and emails or make pull requests. Oh, I'm totally not arguing for cutting off other capabilities, I like tool use and find it to be as useful as the next person! Just that the shell tools that will see A LOT of usage have additional guardrails added on top of them, because it's inevitable that sooner or later any given LLM will screw up and pipe the wrong thing in the wrong command - since you already hear horror stories about devs whose entire machines get wiped. Not everyone has proper backups (even though they totally should)!
walthamstow 7mo ago
Claude has told me that its Grep tool does use rg under the hood, but I constantly find it using the Bash tool with grep
- giancarlostoro 7mo ago
  When I tell it to use rg it goes much faster than it using grep. I really don't understand why its slower with grep.
troupo 7mo ago
> Claude should be shipped by a custom implementation of And when that fails for some reason it will happily write and execute a Python script bypassing all those custom tools
throwaway2027 7mo ago
All of which is useless when it just starts using big blocks of python instead. You need filesystem sandboxing for the python interpreter too.
- ethanwillis 7mo ago
  What we need is a capabilities based security system. It could write all the python, asm, whatever it wants and it wouldn't matter at all if it was never given a reference to use something it shouldn't.
  - mcv 7mo ago
    Isn't this already possible? Give it its own user account with write access to the project directory and either read access or no access outside it.
    - 100721 7mo ago
      I have been putting my agents on their own, restricted OS-level user accounts for a while. It works really well for everything I do. Admittedly, there’s a little more friction and agent confusion sometimes with this setup, but it’s worth the benefit of having zero worries about permissions and security.
      - jmogly 7mo ago
        Haha, you can already see wheel reinventors in this thread starting to spin their reinvention wheels. Nice stuff, I run my agents in containers.
    - VorpalWay 7mo ago
      Unix permissions is not a capability system though. Capabilities are more like "here is a file descriptor pointing to a directory, you are not capable of referring to anything outside it". So closer to chroot, except you can have several such directory references at the same time. You can always narrow down a capability (get a new capability pointing to a subdirectory or file, or remove the writing capability so it is read only) but never make it more broad. In a system designed for this it will be used for everything, not just file system. You might have capabilities related to network connections, or IPC to other processes, etc. The latter is especially attractive in microkernel based OSes. (Speaking of which, Redox OS seems to be experimenting with this, just saw an article today about that.) See also https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security
lxgr 7mo ago
> a custom implementation of "rm" that Anthropic can add guardrails to Wrong layer. You want the deletion to actually be impossible from a privilege perspective, not be made practically harder to the entity that shouldn't delete something. Claude definitely knows how to reimplement `rm`.
nananana9 7mo ago
Oh, rm failed, since we're running in a weird environment! Let me retry with `bash -c "/usr/bin/rm -rf *"`!
- giancarlostoro 7mo ago
  Ideally they control the harness and should be able to stop Claude from running any shell willy nilly.
  - estimator7292 7mo ago
    Thus defeating the purpose of a custom "rm"
torginus 7mo ago
Why cant you ship with OverlayFS which actually enforces these restrictions? I have seen the AI break out of (my admittedly flimsy) guards, like doing simply safepath/../../stuff or something even more convoluted like symlinks.

oefrha 7mo ago

You can define your own rm shell alias/function and it will use that. I also have cp/mv aliases that forces -i to avoid accidental clobbering and it confuses Claude to no end (it uses cp/mv rare enough—rarer than it should, really—that I don’t bother wasting memory tokens on it).

d1sxeyes 7mo ago
I did this, Claude detected it and decided to run /bin/rm directly.
- cestivan 7mo ago
  [dead]
- cogogo 7mo ago
  This is terrifying. I have not used agents because I do not have a sandbox machine I do not care about. Am I crazy to worry about a sandboxed agent running on my home network? Anyone experienced anything weird by doing that?
  - oefrha 7mo ago
    Don’t dangerously skip permissions and actually read commands when you get prompted and you’re fine.
    - d1sxeyes 7mo ago
      Yeah, I actually have both an alias for `rm` and a custom seatbelt sandbox which means the agent can only delete stuff within the directory it’s working in, so wasn’t an issue, was just fun to watch it say “hm, that doesn’t seem to work. Looks like the user has aliased rm. I’ll just go ahead and work around it”
      - cruffle_duffle 7mo ago
        Hah… I’ve seen Claude happily and very cleverly find ways to escape its sandbox. It’s like some kind of arms race between the model and its designers.

eru 7mo ago

> It's really surprising they don't just tweak what Claude uses and lock it down to where it cannot be harmful. Ensure it only ever calls tooling Claude Code provides. That would make it far less useful in general.

KronisLV 7mo ago
Maybe Anthropic (or some collection of the large AI orgs, like OpenAI and Anthropic and Google coming together) should apply patches on top of (or fork altogether) the coreutils and whatever you normally get in a userland - a bit like what you get in Git Bash on Windows, just with: 1) more guardrails in place 2) maybe more useful error messages that would help LLMs 3) no friction with needing to get any patches upstreamed External tool calling should still be an option ofc, but having utilities that are usable just like what's in the training data, but with more security guarantees and more useful output that makes what's going on immediately obvious would be great.
- eru 7mo ago
  So for me, it's really, really useful for Claude to be able to send Slack messages and emails or make pull requests. But that's also the most damaging actions it could take. Everything on my computer is backed up, but if Claude insults my boss, that would be worse.
  - KronisLV 7mo ago
    > So for me, it's really, really useful for Claude to be able to send Slack messages and emails or make pull requests. Oh, I'm totally not arguing for cutting off other capabilities, I like tool use and find it to be as useful as the next person! Just that the shell tools that will see A LOT of usage have additional guardrails added on top of them, because it's inevitable that sooner or later any given LLM will screw up and pipe the wrong thing in the wrong command - since you already hear horror stories about devs whose entire machines get wiped. Not everyone has proper backups (even though they totally should)!

walthamstow 7mo ago

Claude has told me that its Grep tool does use rg under the hood, but I constantly find it using the Bash tool with grep

giancarlostoro 7mo ago
When I tell it to use rg it goes much faster than it using grep. I really don't understand why its slower with grep.

troupo 7mo ago

> Claude should be shipped by a custom implementation of And when that fails for some reason it will happily write and execute a Python script bypassing all those custom tools

throwaway2027 7mo ago

All of which is useless when it just starts using big blocks of python instead. You need filesystem sandboxing for the python interpreter too.

ethanwillis 7mo ago
What we need is a capabilities based security system. It could write all the python, asm, whatever it wants and it wouldn't matter at all if it was never given a reference to use something it shouldn't.
- mcv 7mo ago
  Isn't this already possible? Give it its own user account with write access to the project directory and either read access or no access outside it.
  - 100721 7mo ago
    I have been putting my agents on their own, restricted OS-level user accounts for a while. It works really well for everything I do. Admittedly, there’s a little more friction and agent confusion sometimes with this setup, but it’s worth the benefit of having zero worries about permissions and security.
    - jmogly 7mo ago
      Haha, you can already see wheel reinventors in this thread starting to spin their reinvention wheels. Nice stuff, I run my agents in containers.
  - VorpalWay 7mo ago
    Unix permissions is not a capability system though. Capabilities are more like "here is a file descriptor pointing to a directory, you are not capable of referring to anything outside it". So closer to chroot, except you can have several such directory references at the same time. You can always narrow down a capability (get a new capability pointing to a subdirectory or file, or remove the writing capability so it is read only) but never make it more broad. In a system designed for this it will be used for everything, not just file system. You might have capabilities related to network connections, or IPC to other processes, etc. The latter is especially attractive in microkernel based OSes. (Speaking of which, Redox OS seems to be experimenting with this, just saw an article today about that.) See also https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security

lxgr 7mo ago

> a custom implementation of "rm" that Anthropic can add guardrails to Wrong layer. You want the deletion to actually be impossible from a privilege perspective, not be made practically harder to the entity that shouldn't delete something. Claude definitely knows how to reimplement `rm`.

nananana9 7mo ago

Oh, rm failed, since we're running in a weird environment! Let me retry with `bash -c "/usr/bin/rm -rf *"`!

giancarlostoro 7mo ago
Ideally they control the harness and should be able to stop Claude from running any shell willy nilly.
- estimator7292 7mo ago
  Thus defeating the purpose of a custom "rm"

torginus 7mo ago

Why cant you ship with OverlayFS which actually enforces these restrictions? I have seen the AI break out of (my admittedly flimsy) guards, like doing simply safepath/../../stuff or something even more convoluted like symlinks.