6 ms·

This should give a second life to Test-Driven Development. One of the under-appreciated wisdoms of TDD is that there is a complexity asymmetry in many problems

This should give a second life to Test-Driven Development. One of the under-appreciated wisdoms of TDD is that there is a complexity asymmetry in many problems between finding a solution and (fully or partially) verifying it. Examples of asymmetric problems: inverting matrices, sorting an array, computing a function's gradient, compressing a byte stream, etc. Human writes the easier part -- the test suite, the language model writes the harder part -- the solution. This can be a net gain in productivity.

fodkodrasz 3y ago
Finally something will kill the last somewhat satisfying part of the job finally, for the greater profit of the corporate overlords!
BaseballPhysics 3y ago
The problem is without a formal definition of the program semantics, you run the risk of overfitting or uncovered behaviors that, for a human developer who understands the intent of the program, would be implicit. And given how hard formal verification is, I don't know that you'll ever get away with not having to manually check these programs, at which point I question just how much productivity you've gained. It's kinda like self-driving cars: when they work, they work great. But when they fail, they fail in ways a human never would, and therefore a human struggles to anticipate or trust their behaviour. That said, I'm waiting to see the rise of programming languages designed with LLMs in mind, where a human could use contract oriented programming or similar (think: Ada) combined with TDD methods to more formally specify the problem that an LLM is being asked to solve.
- hellodanylo 3y ago
  > I question just how much productivity you've gained. Me too. It's an empirical question to be answered by those who will dare to try. > It's kinda like self-driving cars Strong disagree. Yes, neural nets are blackboxes, but the generated code can be idiomatic, modular, easy to inspect with a debugger, etc. > more formally specify the problem that an LLM is being asked to solve. That would be a great direction to explore.
  - BaseballPhysics 3y ago
    > Strong disagree. Yes, neural nets are blackboxes, but the generated code can be idiomatic, modular, easy to inspect with a debugger, etc. I think you missed my point. If I'm inspecting code from another human, I'm going to make assumptions about the kinds of errors they're gonna make. There's probably obvious dumb stuff I won't look for because a human would never typically make certain classes of mistake. They're the self-driving car equivalent of driving into the back of a stopped semi truck because it was mistaken for a billboard, an error no human of sound mind and body would make. So if I'm inspecting code written by a computer, I'll either 1) make those same assumptions and then run the risk of missing unexpected problems in the code, or 2) I'm gonna be overly cautious (because I don't trust the machine) and will examine the code with a fine tooth comb, which will take a great deal more time.
    - travisjungroth 3y ago
      Based on my experience with Autopilot and Copilot, I think this is way less of a problem in code. You can put code mistakes on a gradient, from subtle to obvious. Obvious bugs are like when the LLM finds a pattern and repeats it for 100 lines. Subtle mistakes are like misnaming a variable so you use one left over from earlier, not the correct one. Obvious mistakes are easy to catch because they’re obvious. The LLM makes more of those. I think because of the way LLMs work, I have never seen Copilot make a subtle mistake that I wouldn’t expect of a person. People are so good at making surprising bugs, it’s really hard for Copilot to beat it.
darepublic 3y ago
the LLM should write the test suite too, from the description of the app
ramesh31 3y ago
>Human writes the easier part -- the test suite, the language model writes the harder part -- the solution. This can be a net gain in productivity. I've been doing just this with Llama 2 local lately, and I'm convinced that anyone who isn't will be a dinosaur in a couple years. Fact is, these things are damn good (at JS at least) when dealing with pure unit-testable functions. To the point that I can accept practically any solution it comes up with will be better than mine. Everything is still verified of course, but the days of writing algorithms from scratch are basically over for me.
- hk__2 3y ago
  How do you avoid overfitting and how do you test performance?
  - travisjungroth 3y ago
    It would be quite hard to overfit with property based tests.
- hughesjj 3y ago
  How readable is the code? Does it follow SOLID design and the likes?
  - ramesh31 3y ago
    >How readable is the code? Does it follow SOLID design and the likes? More readable and better commented than my own. The thing is, this isn't generating any kind of "compiler output" type code that you're used to. It writes exactly as a person would. Here's an example I just ran on llama2-7b-chat on my Macbook: ##PROMPT Llama: My name is Llama. I will write a Javascript function that implements the "emailTest" function, given the following test case that it is expected to pass. I am sure to always fully explain all of my code, and include comments to provide context. describe('it takes a string as input, validates that it is an email address, and returns the host name of the email address.', () => { const testVal = emailTest('test@google.com'); expect(testVal).toBe('google.com') }); ##END PROMPT ##RESPONSE Llama: Here are the test cases: test('it takes a string as input, validates that it is an email address, and returns the host name of the email address.', () => { const testVal = emailTest('test@google.com'); expect(testVal).toBe('google.com') }) test('it should validate email addresses with different domains', () => { const testVal1 = emailTest('hello@example.com'); const testVal2 = emailTest('hello@example.net'); expect(testVal1).toBe('example.com') expect(testVal2).toBe('example.net') }) test('it should validate email addresses with special characters', () => { const testVal3 = emailTest('hello@$%^&*()_!'); expect(testVal3).toBe('example.com') }) test('it should validate email addresses with typos', () => { const testVal4 = emailTest('hellow@example.com'); expect(testVal4).toBe('example.com') }); Here is the function that I have written to implement the emailTest function: function emailTest(input) { const emailRegex = /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/; if (emailRegex.test(input)) { // If the input is a valid email address, return the host name of the email address const hostName = input.split('@')[1]; return hostName; } else { // Otherwise, return an error message indicating that the input is not a valid email address return 'Invalid email address'; } } Can you please provide feedback on my implementation? Specifically, I would like to know if there are any improvements I could make to the function's performance or accuracy. ##END RESPONSE And from there you can have a conversation with it, and ask it to fix anything that's mistaken. You can also have it extend your test suite with fuzzing and negative cases. This is obviously a trivial example, but you get the idea. I haven't been able to stump it yet.
michaelfeathers 3y ago
I think people aren't getting yet is that this generalizable beyond software. We can do this with LLM and anything that can be designed. https://michaelfeathers.silvrback.com/prompt-hoisting-for-gpt-based-code-generation https://michaelfeathers.silvrback.com/prompt-hoisting-for-gp...
zitterbewegung 3y ago
And with code already that exists we can have the AI help us with writing the tests for TDD. Many large companies have huge amounts of code that are essential to their operation but changing it is difficult and maintaining it to replace it with a new system is difficult or even impossible since it’s very difficult to even know what it does.
- visarga 3y ago
  Using AI to automate TDD is a great idea but unfortunately AIs can get stuck and we are far from solving this problem. Many pieces of information to exactly manipulate while LLMs are all soft and fuzzy.
- deleted 3y ago
  [deleted]

fodkodrasz 3y ago

Finally something will kill the last somewhat satisfying part of the job finally, for the greater profit of the corporate overlords!

BaseballPhysics 3y ago

The problem is without a formal definition of the program semantics, you run the risk of overfitting or uncovered behaviors that, for a human developer who understands the intent of the program, would be implicit. And given how hard formal verification is, I don't know that you'll ever get away with not having to manually check these programs, at which point I question just how much productivity you've gained. It's kinda like self-driving cars: when they work, they work great. But when they fail, they fail in ways a human never would, and therefore a human struggles to anticipate or trust their behaviour. That said, I'm waiting to see the rise of programming languages designed with LLMs in mind, where a human could use contract oriented programming or similar (think: Ada) combined with TDD methods to more formally specify the problem that an LLM is being asked to solve.

hellodanylo 3y ago
> I question just how much productivity you've gained. Me too. It's an empirical question to be answered by those who will dare to try. > It's kinda like self-driving cars Strong disagree. Yes, neural nets are blackboxes, but the generated code can be idiomatic, modular, easy to inspect with a debugger, etc. > more formally specify the problem that an LLM is being asked to solve. That would be a great direction to explore.
- BaseballPhysics 3y ago
  > Strong disagree. Yes, neural nets are blackboxes, but the generated code can be idiomatic, modular, easy to inspect with a debugger, etc. I think you missed my point. If I'm inspecting code from another human, I'm going to make assumptions about the kinds of errors they're gonna make. There's probably obvious dumb stuff I won't look for because a human would never typically make certain classes of mistake. They're the self-driving car equivalent of driving into the back of a stopped semi truck because it was mistaken for a billboard, an error no human of sound mind and body would make. So if I'm inspecting code written by a computer, I'll either 1) make those same assumptions and then run the risk of missing unexpected problems in the code, or 2) I'm gonna be overly cautious (because I don't trust the machine) and will examine the code with a fine tooth comb, which will take a great deal more time.
  - travisjungroth 3y ago
    Based on my experience with Autopilot and Copilot, I think this is way less of a problem in code. You can put code mistakes on a gradient, from subtle to obvious. Obvious bugs are like when the LLM finds a pattern and repeats it for 100 lines. Subtle mistakes are like misnaming a variable so you use one left over from earlier, not the correct one. Obvious mistakes are easy to catch because they’re obvious. The LLM makes more of those. I think because of the way LLMs work, I have never seen Copilot make a subtle mistake that I wouldn’t expect of a person. People are so good at making surprising bugs, it’s really hard for Copilot to beat it.

darepublic 3y ago

the LLM should write the test suite too, from the description of the app

ramesh31 3y ago

>Human writes the easier part -- the test suite, the language model writes the harder part -- the solution. This can be a net gain in productivity. I've been doing just this with Llama 2 local lately, and I'm convinced that anyone who isn't will be a dinosaur in a couple years. Fact is, these things are damn good (at JS at least) when dealing with pure unit-testable functions. To the point that I can accept practically any solution it comes up with will be better than mine. Everything is still verified of course, but the days of writing algorithms from scratch are basically over for me.

hk__2 3y ago
How do you avoid overfitting and how do you test performance?
- travisjungroth 3y ago
  It would be quite hard to overfit with property based tests.
hughesjj 3y ago
How readable is the code? Does it follow SOLID design and the likes?
- ramesh31 3y ago
  >How readable is the code? Does it follow SOLID design and the likes? More readable and better commented than my own. The thing is, this isn't generating any kind of "compiler output" type code that you're used to. It writes exactly as a person would. Here's an example I just ran on llama2-7b-chat on my Macbook: ##PROMPT Llama: My name is Llama. I will write a Javascript function that implements the "emailTest" function, given the following test case that it is expected to pass. I am sure to always fully explain all of my code, and include comments to provide context. describe('it takes a string as input, validates that it is an email address, and returns the host name of the email address.', () => { const testVal = emailTest('test@google.com'); expect(testVal).toBe('google.com') }); ##END PROMPT ##RESPONSE Llama: Here are the test cases: test('it takes a string as input, validates that it is an email address, and returns the host name of the email address.', () => { const testVal = emailTest('test@google.com'); expect(testVal).toBe('google.com') }) test('it should validate email addresses with different domains', () => { const testVal1 = emailTest('hello@example.com'); const testVal2 = emailTest('hello@example.net'); expect(testVal1).toBe('example.com') expect(testVal2).toBe('example.net') }) test('it should validate email addresses with special characters', () => { const testVal3 = emailTest('hello@$%^&*()_!'); expect(testVal3).toBe('example.com') }) test('it should validate email addresses with typos', () => { const testVal4 = emailTest('hellow@example.com'); expect(testVal4).toBe('example.com') }); Here is the function that I have written to implement the emailTest function: function emailTest(input) { const emailRegex = /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/; if (emailRegex.test(input)) { // If the input is a valid email address, return the host name of the email address const hostName = input.split('@')[1]; return hostName; } else { // Otherwise, return an error message indicating that the input is not a valid email address return 'Invalid email address'; } } Can you please provide feedback on my implementation? Specifically, I would like to know if there are any improvements I could make to the function's performance or accuracy. ##END RESPONSE And from there you can have a conversation with it, and ask it to fix anything that's mistaken. You can also have it extend your test suite with fuzzing and negative cases. This is obviously a trivial example, but you get the idea. I haven't been able to stump it yet.

michaelfeathers 3y ago

I think people aren't getting yet is that this generalizable beyond software. We can do this with LLM and anything that can be designed. https://michaelfeathers.silvrback.com/prompt-hoisting-for-gpt-based-code-generation https://michaelfeathers.silvrback.com/prompt-hoisting-for-gp...

zitterbewegung 3y ago

And with code already that exists we can have the AI help us with writing the tests for TDD. Many large companies have huge amounts of code that are essential to their operation but changing it is difficult and maintaining it to replace it with a new system is difficult or even impossible since it’s very difficult to even know what it does.

visarga 3y ago
Using AI to automate TDD is a great idea but unfortunately AIs can get stuck and we are far from solving this problem. Many pieces of information to exactly manipulate while LLMs are all soft and fuzzy.
deleted 3y ago
[deleted]