3 ms·

> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph. Soun

> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph. Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.

xyzsparetimexyz 2mo ago
it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
- olmo23 2mo ago
  sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
- ballon_monkey 2mo ago
  If you're building your own system this is an easy problem to solve.
- vidarh 2mo ago
  Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
  - Terr_ 2mo ago
    Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
  - TuringTest 2mo ago
    > Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself. In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results. There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes. Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems. [1] https://en.wikipedia.org/wiki/Viable_system_model https://en.wikipedia.org/wiki/Viable_system_model
    - xyzsparetimexyz 2mo ago
      But I don't want to have to trust the system to not do the bad thing when I'm away
      - TuringTest 2mo ago
        You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
    - vidarh 2mo ago
      > That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself. No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time. I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
- ngruhn 2mo ago
  Tell me about. Well it should have MCP access in plan mode to lookup backstage docs, right?. Agent proceeds to launch playwright sessions...
- wongarsu 2mo ago
  The simplest design Anthropic could come up with: Have a haiku model quickly judge if a command is harmless or potentially interesting, route all the interesting commands to a sonnet agent judging it properly. Running two full llms sounds slightly insane, but it's apparently the best we can do (and is reasonably cheap since they don't need a lot of context)
- _whiteCaps_ 2mo ago
  My company's rules deny permissions for Read("secret") but not "mv secret not-secret" :-/
thousand_nights 2mo ago
people want fuzzy analog machines with digital controls, it's impossible
- fragmede 2mo ago
  I don't think they actually wanted to. That's just where the technology is, unfortunately.
  - perching_aix 2mo ago
    How do you mean?
    - fragmede 2mo ago
      > people want fuzzy analog machines with digital controls, it's impossible people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
      - topaz0 2mo ago
        We absolutely have digital machines with digital controls, they're called computers, and the controls are called programming languages etc. The reason people prefer llms over programming is exactly because it lets them specify their program fuzzily, i.e. it lets them avoid going to the trouble of specifying enough detail to make it deterministic.
        fragmede 2mo ago
        We have to make the distinction between specification and execution. LLM programming is great because I can underspecify and the machine will infer the rest. Usually reasonably. People want fuzzy specification but deterministic outputs
spike021 2mo ago
I can't even get Claude to stop writing python to parse json instead of using jq despite baking it into agent memory and skills.
- NegativeLatency 2mo ago
  Try your main agents file, not quite the same thing but I’ve been able to get mine to use better tools most of the time https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools
- xnorswap 2mo ago
  I've resorted to uninstalling python to stop it writing python scripts.

xyzsparetimexyz 2mo ago

it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around

olmo23 2mo ago
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
ballon_monkey 2mo ago
If you're building your own system this is an easy problem to solve.
vidarh 2mo ago
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
- Terr_ 2mo ago
  Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
- TuringTest 2mo ago
  > Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself. In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results. There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes. Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems. [1] https://en.wikipedia.org/wiki/Viable_system_model https://en.wikipedia.org/wiki/Viable_system_model
  - xyzsparetimexyz 2mo ago
    But I don't want to have to trust the system to not do the bad thing when I'm away
    - TuringTest 2mo ago
      You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
  - vidarh 2mo ago
    > That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself. No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time. I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
ngruhn 2mo ago
Tell me about. Well it should have MCP access in plan mode to lookup backstage docs, right?. Agent proceeds to launch playwright sessions...
wongarsu 2mo ago
The simplest design Anthropic could come up with: Have a haiku model quickly judge if a command is harmless or potentially interesting, route all the interesting commands to a sonnet agent judging it properly. Running two full llms sounds slightly insane, but it's apparently the best we can do (and is reasonably cheap since they don't need a lot of context)
_whiteCaps_ 2mo ago
My company's rules deny permissions for Read("secret") but not "mv secret not-secret" :-/

thousand_nights 2mo ago

people want fuzzy analog machines with digital controls, it's impossible

fragmede 2mo ago
I don't think they actually wanted to. That's just where the technology is, unfortunately.
- perching_aix 2mo ago
  How do you mean?
  - fragmede 2mo ago
    > people want fuzzy analog machines with digital controls, it's impossible people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
    - topaz0 2mo ago
      We absolutely have digital machines with digital controls, they're called computers, and the controls are called programming languages etc. The reason people prefer llms over programming is exactly because it lets them specify their program fuzzily, i.e. it lets them avoid going to the trouble of specifying enough detail to make it deterministic.
      - fragmede 2mo ago
        We have to make the distinction between specification and execution. LLM programming is great because I can underspecify and the machine will infer the rest. Usually reasonably. People want fuzzy specification but deterministic outputs

spike021 2mo ago

I can't even get Claude to stop writing python to parse json instead of using jq despite baking it into agent memory and skills.

NegativeLatency 2mo ago
Try your main agents file, not quite the same thing but I’ve been able to get mine to use better tools most of the time https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools
xnorswap 2mo ago
I've resorted to uninstalling python to stop it writing python scripts.