3 ms·

This is the future: A Software Library with No Code

38 points by ulrischa 9mo ago

simonw 9mo ago
I've been exploring this pattern recently too. Giving current coding agents an existing conformance or test suite and telling them to keep writing code unto the tests pass is astonishingly effective. I've now got a JavaScript interpreter and a WebAssembly runtime written in Python, built by Claude Code for web run from my phone.
NitpickLawyer 9mo ago
Ok, I really liked this! A cool concept that would've been sci-fi ~5 years ago (not this specific library, but the idea of spec->code in any language, 0 human intervention). I can see a collection of utilities "specced out" and then shared as specs + tests. Perhaps for harder stuff even some spec formal verification + tests? (no idea how feasible this is, but it's something to look into) I had no doubts claude/codex/gemini could handle this, but I was surprised that even an open-source model (glm4.7) did it 1 shot (just c/p the prompt from Install.md and came back 10 min later): ============================= 123 passed in 0.18s ============================== Test Results: All 123 tests from tests.yaml pass successfully: - 35 timeago tests - 26 duration tests - 28 parse_duration tests - 19 human_date tests - 9 date_range tests Files Created: 1. /workspaces/glmcode/whenwords/src/__init__.py - Library implementation 2. /workspaces/glmcode/whenwords/test_whenwords.py - Test file generated from tests.yaml 3. /workspaces/glmcode/whenwords/usage.md - Usage documentation Crunched for 11m 38s ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Showing detailed transcript · ctrl+o to toggle I mean, it's a "toy" library, but the concept is so cool! And the fact that an open, locally hostable model can do it 1shot is insane.
roxolotl 9mo ago
Why wouldn't we just specify these things with programming languages though? The purpose of a programming language is to enable precise specification of computation. SPEC.md is 497 lines, the tests another 500. I don't understand what we'd gain from taking a spec which is imprecise and tests, which seemingly are arbitrary and cover the same cases multiple times, putting them into an LLM and recreating a version of the computation you desire. If an LLM can do this with English and arbitrary test cases why wouldn't you pick a programming language and specific test cases? This would give you significantly more repeatability and consistency all while probably having less overall lines of code.
- simonw 9mo ago
  You can define them in a structured way that's not tied to a specific programming language. Imagine a test suite that's entirely YAML inputs and outputs, or JSON, or even CSV. The key idea is to have one test suite/specification that multiple implantations in different languages can share.
  - roxolotl 9mo ago
    What is the advantage of that over programming languages though? At some point you’re just creating a new specification language which needs to be learned. If an LLM can go from English spec to Python unit tests why not just start with, or at least distribute, Python unit tests. A programming language will allow you to be significantly more correct and consistent than English.
    - simonw 9mo ago
      Because if the tests are in Python the LLM still has to convert them from Python to Ruby or whatever, which leaves room for mistakes to creep in. If the tests are in YAML it doesn't need to convert them at all. It can write a new test harness in the new language and run against those existing, deterministic tests.
      - roxolotl 9mo ago
        My point is that to create a specification you need to use a formal language of some kind. In this example they created a new yaml based specification language. Why do that vs use a well documented existing formal language the LLM knows well like Python. The translation is either yaml -> new language or Python -> new language. The translation is happening in both cases. The advantage I can think of is it would might be more human readable but Python is damn close to pseudocode. It’ll likely always be a bit annoying to write because it has to be a formal language.
        simonw 9mo ago
pseudohadamard 9mo ago
[flagged]
seridarivus13 9mo ago
[dead]
skybrian 9mo ago
> If the customer gets stuck on an issue with their own generated codebase, how do we have a hope of finding the problem? Effectively, the coding agent has to provide front-line support. They ask the coding agent to diagnose the bug and either fix it directly, or generate a bug report and send it upstream. It seems like a better idea to have a downstream maintainer generate and maintain the language-specific code? If you're providing enterprise support, maybe that downstream maintainer is you.

38 points by ulrischa 9mo ago

I've been exploring this pattern recently too. Giving current coding agents an existing conformance or test suite and telling them to keep writing code unto the tests pass is astonishingly effective. I've now got a JavaScript interpreter and a WebAssembly runtime written in Python, built by Claude Code for web run from my phone.

NitpickLawyer 9mo ago

Ok, I really liked this! A cool concept that would've been sci-fi ~5 years ago (not this specific library, but the idea of spec->code in any language, 0 human intervention). I can see a collection of utilities "specced out" and then shared as specs + tests. Perhaps for harder stuff even some spec formal verification + tests? (no idea how feasible this is, but it's something to look into) I had no doubts claude/codex/gemini could handle this, but I was surprised that even an open-source model (glm4.7) did it 1 shot (just c/p the prompt from Install.md and came back 10 min later): ============================= 123 passed in 0.18s ============================== Test Results: All 123 tests from tests.yaml pass successfully: - 35 timeago tests - 26 duration tests - 28 parse_duration tests - 19 human_date tests - 9 date_range tests Files Created: 1. /workspaces/glmcode/whenwords/src/__init__.py - Library implementation 2. /workspaces/glmcode/whenwords/test_whenwords.py - Test file generated from tests.yaml 3. /workspaces/glmcode/whenwords/usage.md - Usage documentation Crunched for 11m 38s ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Showing detailed transcript · ctrl+o to toggle I mean, it's a "toy" library, but the concept is so cool! And the fact that an open, locally hostable model can do it 1shot is insane.

roxolotl 9mo ago

Why wouldn't we just specify these things with programming languages though? The purpose of a programming language is to enable precise specification of computation. SPEC.md is 497 lines, the tests another 500. I don't understand what we'd gain from taking a spec which is imprecise and tests, which seemingly are arbitrary and cover the same cases multiple times, putting them into an LLM and recreating a version of the computation you desire. If an LLM can do this with English and arbitrary test cases why wouldn't you pick a programming language and specific test cases? This would give you significantly more repeatability and consistency all while probably having less overall lines of code.

simonw 9mo ago
You can define them in a structured way that's not tied to a specific programming language. Imagine a test suite that's entirely YAML inputs and outputs, or JSON, or even CSV. The key idea is to have one test suite/specification that multiple implantations in different languages can share.
- roxolotl 9mo ago
  What is the advantage of that over programming languages though? At some point you’re just creating a new specification language which needs to be learned. If an LLM can go from English spec to Python unit tests why not just start with, or at least distribute, Python unit tests. A programming language will allow you to be significantly more correct and consistent than English.
  - simonw 9mo ago
    Because if the tests are in Python the LLM still has to convert them from Python to Ruby or whatever, which leaves room for mistakes to creep in. If the tests are in YAML it doesn't need to convert them at all. It can write a new test harness in the new language and run against those existing, deterministic tests.
    - roxolotl 9mo ago
      My point is that to create a specification you need to use a formal language of some kind. In this example they created a new yaml based specification language. Why do that vs use a well documented existing formal language the LLM knows well like Python. The translation is either yaml -> new language or Python -> new language. The translation is happening in both cases. The advantage I can think of is it would might be more human readable but Python is damn close to pseudocode. It’ll likely always be a bit annoying to write because it has to be a formal language.
      - simonw 9mo ago

pseudohadamard 9mo ago

[flagged]

seridarivus13 9mo ago

[dead]

skybrian 9mo ago

> If the customer gets stuck on an issue with their own generated codebase, how do we have a hope of finding the problem? Effectively, the coding agent has to provide front-line support. They ask the coding agent to diagnose the bug and either fix it directly, or generate a bug report and send it upstream. It seems like a better idea to have a downstream maintainer generate and maintain the language-specific code? If you're providing enterprise support, maybe that downstream maintainer is you.