4 ms·

Why wouldn't we just specify these things with programming languages though? The purpose of a programming language is to enable precise specification of computa

Why wouldn't we just specify these things with programming languages though? The purpose of a programming language is to enable precise specification of computation. SPEC.md is 497 lines, the tests another 500. I don't understand what we'd gain from taking a spec which is imprecise and tests, which seemingly are arbitrary and cover the same cases multiple times, putting them into an LLM and recreating a version of the computation you desire. If an LLM can do this with English and arbitrary test cases why wouldn't you pick a programming language and specific test cases? This would give you significantly more repeatability and consistency all while probably having less overall lines of code.

simonw 9mo ago
You can define them in a structured way that's not tied to a specific programming language. Imagine a test suite that's entirely YAML inputs and outputs, or JSON, or even CSV. The key idea is to have one test suite/specification that multiple implantations in different languages can share.
- roxolotl 9mo ago
  What is the advantage of that over programming languages though? At some point you’re just creating a new specification language which needs to be learned. If an LLM can go from English spec to Python unit tests why not just start with, or at least distribute, Python unit tests. A programming language will allow you to be significantly more correct and consistent than English.
  - simonw 9mo ago
    Because if the tests are in Python the LLM still has to convert them from Python to Ruby or whatever, which leaves room for mistakes to creep in. If the tests are in YAML it doesn't need to convert them at all. It can write a new test harness in the new language and run against those existing, deterministic tests.
    - roxolotl 9mo ago
      My point is that to create a specification you need to use a formal language of some kind. In this example they created a new yaml based specification language. Why do that vs use a well documented existing formal language the LLM knows well like Python. The translation is either yaml -> new language or Python -> new language. The translation is happening in both cases. The advantage I can think of is it would might be more human readable but Python is damn close to pseudocode. It’ll likely always be a bit annoying to write because it has to be a formal language.
      - simonw 9mo ago
        There's no translation from YAML to a different language. The YAML describes the tests - like this file here: https://github.com/dbreunig/whenwords/blob/main/tests.yaml https://github.com/dbreunig/whenwords/blob/main/tests.yaml Snippet: - name: "5 hours ago" input: { timestamp: 1704049200, reference: 1704067200 } output: "5 hours ago" - name: "21 hours ago" input: { timestamp: 1703991600, reference: 1704067200 } output: "21 hours ago" When told "use red/green TDD to write code for this in Ruby", a coding agent like Claude Code will write a test harness in Ruby that loops through all of those YAML tests, run it and watch it fail, then write just enough Ruby that the tests pass.
        roxolotl 9mo ago
        Yea I guess we're having a definitional disagreement here. To be clear I think this is a good idea and the work you've done using tests from projects to have agents translate libraries is awesome. But to me clearly that YAML snippet you provided is a specification which needs to be translated to Ruby as much as Python would. If the equivalent Python is: def test_timeago_5_hours_ago(self): self.assertEqual(timeago(1704049200, 1704067200)), "5 hours ago") def test_timeago_21_hours_ago(self): self.assertEqual(timeago(1703991600, 1704067200)), "21 hours ago") The YAML is no more clear than the Python, nor closer to Ruby. Honestly I think it's less clear as a human reading it because it's hard to tell which function is being tested in context of a specific test case. I guess it's possible Claude is better at working with the YAML than the Python but that would be a coincidence I think.
- layer8 9mo ago
  For a formally (in other words, reliably) verifiable implementation, you not only need a formal specification for the library, but also a formal specification of each targeted programming language. And we largely don’t have those.
- Kerrick 9mo ago
  FitNesse?

simonw 9mo ago

You can define them in a structured way that's not tied to a specific programming language. Imagine a test suite that's entirely YAML inputs and outputs, or JSON, or even CSV. The key idea is to have one test suite/specification that multiple implantations in different languages can share.

roxolotl 9mo ago
What is the advantage of that over programming languages though? At some point you’re just creating a new specification language which needs to be learned. If an LLM can go from English spec to Python unit tests why not just start with, or at least distribute, Python unit tests. A programming language will allow you to be significantly more correct and consistent than English.
- simonw 9mo ago
  Because if the tests are in Python the LLM still has to convert them from Python to Ruby or whatever, which leaves room for mistakes to creep in. If the tests are in YAML it doesn't need to convert them at all. It can write a new test harness in the new language and run against those existing, deterministic tests.
  - roxolotl 9mo ago
    My point is that to create a specification you need to use a formal language of some kind. In this example they created a new yaml based specification language. Why do that vs use a well documented existing formal language the LLM knows well like Python. The translation is either yaml -> new language or Python -> new language. The translation is happening in both cases. The advantage I can think of is it would might be more human readable but Python is damn close to pseudocode. It’ll likely always be a bit annoying to write because it has to be a formal language.
    - simonw 9mo ago
      There's no translation from YAML to a different language. The YAML describes the tests - like this file here: https://github.com/dbreunig/whenwords/blob/main/tests.yaml https://github.com/dbreunig/whenwords/blob/main/tests.yaml Snippet: - name: "5 hours ago" input: { timestamp: 1704049200, reference: 1704067200 } output: "5 hours ago" - name: "21 hours ago" input: { timestamp: 1703991600, reference: 1704067200 } output: "21 hours ago" When told "use red/green TDD to write code for this in Ruby", a coding agent like Claude Code will write a test harness in Ruby that loops through all of those YAML tests, run it and watch it fail, then write just enough Ruby that the tests pass.
      - roxolotl 9mo ago
        Yea I guess we're having a definitional disagreement here. To be clear I think this is a good idea and the work you've done using tests from projects to have agents translate libraries is awesome. But to me clearly that YAML snippet you provided is a specification which needs to be translated to Ruby as much as Python would. If the equivalent Python is: def test_timeago_5_hours_ago(self): self.assertEqual(timeago(1704049200, 1704067200)), "5 hours ago") def test_timeago_21_hours_ago(self): self.assertEqual(timeago(1703991600, 1704067200)), "21 hours ago") The YAML is no more clear than the Python, nor closer to Ruby. Honestly I think it's less clear as a human reading it because it's hard to tell which function is being tested in context of a specific test case. I guess it's possible Claude is better at working with the YAML than the Python but that would be a coincidence I think.
layer8 9mo ago
For a formally (in other words, reliably) verifiable implementation, you not only need a formal specification for the library, but also a formal specification of each targeted programming language. And we largely don’t have those.
Kerrick 9mo ago
FitNesse?