11 ms·

When I saw the test suite that Peter Norvig created for the program, I immediately thought to myself “what if there was a LLM program that knew how to generate

When I saw the test suite that Peter Norvig created for the program, I immediately thought to myself “what if there was a LLM program that knew how to generate test cases for arbitrary functions?” I think a tool like that even in an early incomplete and imperfect form could help out a lot of people. The first version could take all available test cases as training data. The second one could instead have a curated list of test cases that pass some bar. Update: I thought of a second idea also based on Peter Norvig’s observation: what about an LLM program that adds documentation / comments to the code without changing the code itself? I know that it is a lot easier for me to proofread writing that I have not seen before, so it would help me. Maybe a version would simply allow for selecting which blocks of code need commenting based on lines selected in an IDE?

hgsgm 4y ago
QuickCheck (not NN) generates test cases. GPT can already comment on code.
kubb 4y ago
That's potentially more helpful than writing the code itself. Writing unit tests can take most of the development time.
- mrguyorama 4y ago
  And literally throwing random half junk unit tests at your code will better test it than you writing unit tests that are blind to the problems it might have because you wrote both and both bits of code have the same blind spots. We should probably be developing systems that fuzz all code by default.
  - SlickNixon 4y ago
    In the future, code beyond all human comprehension will be rated for reliability by the number of layers of garbage unit tests it's buried under. Tests for the code, then tests for the tests, then tests for those te-...
  - danuker 4y ago
    I warmly recommend QuickCheck-inspired libraries.
Buttons840 4y ago
How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated. The next step could be to have the AI write code that describes its own reasoning, balancing length of code and precision.
- spawarotti 4y ago
  > I define a few test cases and the AI writes code for a generalized solution How about the AI never writing any code, just training "mini AI" / network that implements the test cases, of course in a generalized way, the way our current AI systems work. We could continue adding test cases for corner cases until the "mini AI" is so good that we no longer can come up with a test case that trips it over. In such future, the skill of being comprehensive tester would be everything, and the only code written by humans would be the test cases.
- avmich 4y ago
  > How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated. I think Barliman does something similar (http://minikanren.org/ http://minikanren.org/)

hgsgm 4y ago

QuickCheck (not NN) generates test cases. GPT can already comment on code.

kubb 4y ago

That's potentially more helpful than writing the code itself. Writing unit tests can take most of the development time.

mrguyorama 4y ago
And literally throwing random half junk unit tests at your code will better test it than you writing unit tests that are blind to the problems it might have because you wrote both and both bits of code have the same blind spots. We should probably be developing systems that fuzz all code by default.
- SlickNixon 4y ago
  In the future, code beyond all human comprehension will be rated for reliability by the number of layers of garbage unit tests it's buried under. Tests for the code, then tests for the tests, then tests for those te-...
- danuker 4y ago
  I warmly recommend QuickCheck-inspired libraries.

Buttons840 4y ago

How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated. The next step could be to have the AI write code that describes its own reasoning, balancing length of code and precision.

spawarotti 4y ago
> I define a few test cases and the AI writes code for a generalized solution How about the AI never writing any code, just training "mini AI" / network that implements the test cases, of course in a generalized way, the way our current AI systems work. We could continue adding test cases for corner cases until the "mini AI" is so good that we no longer can come up with a test case that trips it over. In such future, the skill of being comprehensive tester would be everything, and the only code written by humans would be the test cases.
avmich 4y ago
> How about the other way. I define a few test cases and the AI writes code for a generalized solution. Not just code that regurgitates the test cases, but that generalizes well to unseen cases. You'll notice this is simply the machine learning problem restated. I think Barliman does something similar (http://minikanren.org/ http://minikanren.org/)