4 ms·

> Also, why not run "npm run test" at some point? We have tons of tests. I even have an integration test that crawls the entire fucking app recusrively link-by-

> Also, why not run "npm run test" at some point? We have tons of tests. I even have an integration test that crawls the entire fucking app recusrively link-by-link in a headless browser and reports on JS errors. CLAUDE.md has all the info. I'm a little baffled by this post. The author claims to have "Wrote a comprehensive CLAUDE.md with detailed instructions." and yet didn't have "run the tests" anywhere? I realize this post is going to be a playground for bashing on AI but I just wish the prompt was published or even better, if it's open source let other people try. Seems like the perfect case to throw claude code in a wiggum loop at overnight.

mingus88 8mo ago
Exactly, if Claude is making these types of mistakes, write a better claude.md instead of a blog post. My company uses an obscure DSL with a name shared with a popular OSS project. Claude was worthless because it kept suggesting code in that other language. Well, we wrote an MCP so Claude could run and test its code and reference the language docs. It’s amazing now. It makes mistakes like this post and then just fixes it and tests again.
suddenlybananas 8mo ago
It's super intelligent but it can't be bothered to run tests unless specifically told to?
- simonw 8mo ago
  Personally I prefer my agents not to run random commands on my machine without me telling them to first. Imagine you just cloned some random project from GitHub and fired up Claude Code in that folder, but it turned out to be malicious and running 'npm test' stole all your files.
  - suddenlybananas 8mo ago
    If it's super intelligent, surely it could glance at tests before running them and figure whether it was malicious or not.
    - simonw 8mo ago
      Tests have dependencies. Crawling all of those dependencies to check for malicious code could require inspecting millions of lines of code, if you could even obtain the code. It's also beginning to sound like needing to solve the halting problem.
      - suddenlybananas 8mo ago
        Come on man. You're being unserious here.
        simonw 8mo ago
        I'm really not. You're the one arguing about a "super intelligent" strawman.
        suddenlybananas 8mo ago
        Look, I know you have a lot invested in this project but I don't see why you think it is somehow unreasonable to expect an AI agent to run tests in a repository. You don't need super intelligence for that.
        simonw 8mo ago
        Of course I went agents to run tests in a repository - I do that all the time. I don't want the agent to run tests in a new repository until I've given it the go-ahead to do that.
pllbnk 8mo ago
Your quoted excerpt implies that CLAUDE.md had this information. Having used Claude Code more than enough I have faced so many issues like the blog's author that I could have written a very similar post (I am not an FE dev though).

mingus88 8mo ago

Exactly, if Claude is making these types of mistakes, write a better claude.md instead of a blog post. My company uses an obscure DSL with a name shared with a popular OSS project. Claude was worthless because it kept suggesting code in that other language. Well, we wrote an MCP so Claude could run and test its code and reference the language docs. It’s amazing now. It makes mistakes like this post and then just fixes it and tests again.

suddenlybananas 8mo ago

It's super intelligent but it can't be bothered to run tests unless specifically told to?

simonw 8mo ago
Personally I prefer my agents not to run random commands on my machine without me telling them to first. Imagine you just cloned some random project from GitHub and fired up Claude Code in that folder, but it turned out to be malicious and running 'npm test' stole all your files.
- suddenlybananas 8mo ago
  If it's super intelligent, surely it could glance at tests before running them and figure whether it was malicious or not.
  - simonw 8mo ago
    Tests have dependencies. Crawling all of those dependencies to check for malicious code could require inspecting millions of lines of code, if you could even obtain the code. It's also beginning to sound like needing to solve the halting problem.
    - suddenlybananas 8mo ago
      Come on man. You're being unserious here.
      - simonw 8mo ago
        I'm really not. You're the one arguing about a "super intelligent" strawman.
        suddenlybananas 8mo ago
        Look, I know you have a lot invested in this project but I don't see why you think it is somehow unreasonable to expect an AI agent to run tests in a repository. You don't need super intelligence for that.
        simonw 8mo ago
        Of course I went agents to run tests in a repository - I do that all the time. I don't want the agent to run tests in a new repository until I've given it the go-ahead to do that.

pllbnk 8mo ago

Your quoted excerpt implies that CLAUDE.md had this information. Having used Claude Code more than enough I have faced so many issues like the blog's author that I could have written a very similar post (I am not an FE dev though).