3 ms·

> In fact I'm genuinely struggling to think of any scenario where an agent CLI would need to execute a build artifact. You should look into "testing". It's wh

by qarl2 3mo ago

> In fact I'm genuinely struggling to think of any scenario where an agent CLI would need to execute a build artifact. You should look into "testing". It's when you run the code you just wrote to make sure it works. For example, in LLM harnesses, it's used to ensure the produced code has the desired properties. It is the key element needed to allow agents to continue to work until they have succeeded. Anthropic's harness Claude Code: https://code.claude.com/docs/en/best-practices https://code.claude.com/docs/en/best-practices — "Claude does the work, runs the check, reads the result, and iterates" https://www-cdn.anthropic.com/58284b19e702b49db9302d5b6f135ad8871e7658.pdf https://www-cdn.anthropic.com/58284b19e702b49db9302d5b6f135a... — "autonomous loops where Claude writes code, runs tests, and iterates continuously" https://www.anthropic.com/research/long-running-Claude https://www.anthropic.com/research/long-running-Claude — "Run pytest tests/ -x -q before every commit" https://platform.claude.com/docs/en/agents-and-tools/tool-use/bash-tool https://platform.claude.com/docs/en/agents-and-tools/tool-us... — "Claude edits a file with one tool and requests the command that runs it" https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool https://platform.claude.com/docs/en/agents-and-tools/tool-us... — "writes Python with the file operations sub-tool and runs it with a Bash command" OpenAI's harness Codex: https://openai.com/index/introducing-codex/ https://openai.com/index/introducing-codex/ — "can iteratively run tests until it receives a passing result" https://openai.com/index/introducing-upgrades-to-codex/ https://openai.com/index/introducing-upgrades-to-codex/ — "iterating on its implementation, fixing test failures, and ultimately delivering a successful implementation" Aider harness: https://aider.chat/ https://aider.chat/ — "Automatically lint and test your code every time aider makes changes" https://aider.chat/docs/usage/lint-test.html https://aider.chat/docs/usage/lint-test.html — "run your test suite after each time the AI edits your code" Devin harness: https://cognition.com/blog/introducing-devin https://cognition.com/blog/introducing-devin — "Devin writes and debugs code to run a computer vision model" https://cognition.com/blog/introducing-devin-2-2 https://cognition.com/blog/introducing-devin-2-2 — "Devin runs through your app and sends back screen recordings" https://cognition.com/blog https://cognition.com/blog — "Devin now builds, runs, and tests natively in Windows VMs" SWE-bench harness: https://www.vals.ai/benchmarks/swebench https://www.vals.ai/benchmarks/swebench — "running unit tests against the generated patch" Copilot: https://github.blog/changelog/2025-05-19-github-copilot-coding-agent-in-public-preview/ https://github.blog/changelog/2025-05-19-github-copilot-codi... — "makes changes, and even validates its work with your tests and linter" https://docs.github.com/copilot/concepts/agents/coding-agent/about-coding-agent https://docs.github.com/copilot/concepts/agents/coding-agent... — "how to build, test and validate its changes" https://docs.github.com/copilot/how-tos/agents/copilot-coding-agent/best-practices-for-using-copilot-to-work-on-tasks https://docs.github.com/copilot/how-tos/agents/copilot-codin... — "If Copilot is able to build, test and validate its changes" https://github.blog/ai-and-ml/github-copilot/github-copilot-coding-agent-101-getting-started-with-agentic-workflows-on-github/ https://github.blog/ai-and-ml/github-copilot/github-copilot-... — "coding agent can independently write, run, and test code"

lucideer 3mo ago
You're definitely referring to fully autonomous harnesses as I suspected - these are generally quite well suited to VM deployment (quite possibly in the cloud) if you're concerned with sandboxing as they don't typically run into the same challenges as an interactive LLM UI. For ad hoc task/investigation-oriented fully interactive agent UIs the requirements are more complex & there's all a lot of advantages to running locally on host, but e.g. for builds & testing in that context there's less of a requirement for the LLM to have full control over every execution given user presence. Basically the use-case being discussed here is LLMs-as-assistance/pair programmers/active digital colleagues, rather than fully automated autonomous harnesses (legit but separate use-case).
- qarl2 3mo ago
  And now you shift the goalposts. All of the harnesses I cited above run locally and default to testing the code they generate, despite your inability to conceive of it. It is very obviously the primary use case: automated development interleaved with investigation. Because you need to do both, at the same time, iteratively. Please don't make me paste those URLs again: they demonstrate exactly what you say does not happen.

by qarl2 3mo ago

lucideer 3mo ago

You're definitely referring to fully autonomous harnesses as I suspected - these are generally quite well suited to VM deployment (quite possibly in the cloud) if you're concerned with sandboxing as they don't typically run into the same challenges as an interactive LLM UI. For ad hoc task/investigation-oriented fully interactive agent UIs the requirements are more complex & there's all a lot of advantages to running locally on host, but e.g. for builds & testing in that context there's less of a requirement for the LLM to have full control over every execution given user presence. Basically the use-case being discussed here is LLMs-as-assistance/pair programmers/active digital colleagues, rather than fully automated autonomous harnesses (legit but separate use-case).

qarl2 3mo ago
And now you shift the goalposts. All of the harnesses I cited above run locally and default to testing the code they generate, despite your inability to conceive of it. It is very obviously the primary use case: automated development interleaved with investigation. Because you need to do both, at the same time, iteratively. Please don't make me paste those URLs again: they demonstrate exactly what you say does not happen.