4 ms·

No, OP said he used the Max Opus 4.6. Anyways, I think one area where Codex and Claude Code falls short is that they do not test the changes they made by using

No, OP said he used the Max Opus 4.6. Anyways, I think one area where Codex and Claude Code falls short is that they do not test the changes they made by using the app. In this case, the LLM should ideally render the page in a real browser, and actually click on the buttons to verify. Best if the LLM test it before the changes, and then after so that it is the same. Maybe it should take a screenshot of before the change, then take a screenshot after. And match. I asked why Codex and Claude don't do this here: https://news.ycombinator.com/item?id=46792066 https://news.ycombinator.com/item?id=46792066

threetonesun 9mo ago
Yeah, if you have these tools in place to validate it's changes you can quickly iterate with it to the right results. But think through how it's making UI changes and it becomes obvious quickly why it can make absolutely wrong and terrible guesses about the implementation details, it can't _see_ what it's doing, or interact with it, it's just pattern matching other implementations its seen.
- aurareturn 9mo ago
  Yea, the next breakthrough for Codex or Claude Code would be to actually use/test the app like a real human would during the development process.
  - simonw 9mo ago
    Here's a document produced by Claude Code using my Showboat testing tool this morning to help explore SeaweedFS (a local S3 clone) - it includes trying things out with curl and getting screenshots from Chrome using my Rodney tool: https://github.com/simonw/research/blob/main/seaweedfs-testing/README.md https://github.com/simonw/research/blob/main/seaweedfs-testi...
throwup238 9mo ago
See the /chrome command in Claude code.
mwigdahl 9mo ago
You can easily do this, at least with Claude Code. Ask it to install and use Playwright to confirm rendering and flow. You're correct that it is a failing to not do this. When you do, it definitely helps cut down on bugs. EDIT: Sorry, just noticed you said "real browser". Haven't tried this but Playwright gets you a long way down the road.
- aurareturn 9mo ago
  Will check it out. Looks like there is also chrome-devtools-mcp for Codex.
lenerdenator 9mo ago
FWIW, I've found Playwright tests to be a decent way of getting Claude to do what you're talking about.

threetonesun 9mo ago

Yeah, if you have these tools in place to validate it's changes you can quickly iterate with it to the right results. But think through how it's making UI changes and it becomes obvious quickly why it can make absolutely wrong and terrible guesses about the implementation details, it can't _see_ what it's doing, or interact with it, it's just pattern matching other implementations its seen.

aurareturn 9mo ago
Yea, the next breakthrough for Codex or Claude Code would be to actually use/test the app like a real human would during the development process.
- simonw 9mo ago
  Here's a document produced by Claude Code using my Showboat testing tool this morning to help explore SeaweedFS (a local S3 clone) - it includes trying things out with curl and getting screenshots from Chrome using my Rodney tool: https://github.com/simonw/research/blob/main/seaweedfs-testing/README.md https://github.com/simonw/research/blob/main/seaweedfs-testi...

throwup238 9mo ago

See the /chrome command in Claude code.

mwigdahl 9mo ago

You can easily do this, at least with Claude Code. Ask it to install and use Playwright to confirm rendering and flow. You're correct that it is a failing to not do this. When you do, it definitely helps cut down on bugs. EDIT: Sorry, just noticed you said "real browser". Haven't tried this but Playwright gets you a long way down the road.

aurareturn 9mo ago
Will check it out. Looks like there is also chrome-devtools-mcp for Codex.

lenerdenator 9mo ago

FWIW, I've found Playwright tests to be a decent way of getting Claude to do what you're talking about.