10 ms·

Training an AI to write unit tests (and refactor code to be more testable) would be a real game changer.

riffraff 3y ago
Copilot does a decent job with "dumb" unit tests, e.g. "when the condition X is Y then Z". But it obviously doesn't have knowledge of your domain, and I don't think any LLM could do a perfect job without that.
byby 3y ago
I don't think you get it. The game has already changed. And nobody trained the AI to write unit tests or refactor code. It could do it as emergent behavior. It happened so fast that so many people are in denial and many people aren't even asking the obvious questions. Simply ASK chatGPT to refactor your code to be more unit testable and IT Can do it. Of course it clearly has your typical LLM problems but it is roughly 50% there. The game has changed. If AI gets to 100%, not only will the game be changed, but humans will no longer be part of the game. See what I asked chatGPT: ME: def (x: int): for i in range(x): print(i) Can you refactor the code so it can be more unit testable? CHATGPT: Certainly! To make the code more unit testable, we can refactor it into a function that returns a list of integers instead of directly printing them. Here's the refactored code: python def generate_number_list(x: int) -> List[int]: number_list = [] for i in range(x): number_list.append(i) return number_list In this refactored code, we've created a function called generate_number_list that takes an integer x as input. Instead of printing each number, we use a number_list variable to store the generated numbers by appending them in the loop. Finally, we return the number_list. By returning the list instead of printing the numbers directly, we make the code more testable because we can now write unit tests that assert the correctness of the generated list. If boggles my mind that I see threads of people saying "what if we train LLMs to do this or to do that?" or whatever. Guys, just ask the LLM to do what you want in plain English. It can do it.
- mjburgess 3y ago
  That isn't the game. The game is designing software to requirements. It's writing literature for a new era. It's creating X for A audience with N vauge unspecified needs -- where X is a complex product made of many parts, involving many people, with shifting and changing problems/solutions/requirements. The game was never writing the stack overflow answer -- that was already written.
  - byby 3y ago
    >The game was never writing the stack overflow answer -- that was already written. The problem is this was never a stackoverflow question and there was never an answer for it. Try finding it. The LLM is already playing the game because it came up with that answer which is Fully Correct, Out of Thin Air. Look, clearly the LLM can't play the game as well as a trained adept human, but it's definitely playing the game. >The game is designing software to requirements. It's writing literature for a new era. It's creating X for A audience with N vauge unspecified needs -- where X is a complex product made of many parts, involving many people, with shifting and changing problems/solutions/requirements. It can do all of this. It can talk like you and parrot exactly what your saying and also go into more detail and re-frame your words more eloquently. What you're not getting is that all the things you mentioned the LLM can do in actuality to varying degrees to the point where it is in the "game." and at times it does better than us. Likely, you haven't even tried asking it yet.
    - mjburgess 3y ago
      > Fully Correct, Out of Thin Air I think if you're an expert in an area, this effect is easier to see through. You know where the github repo is, where the library example is, which ebooks there area -- etc. and you're mostly at-ease not using them and just writing the solution yourself. These systems are not "fully correct" and not "out of thin area". They are trained on everything ever digitised, including the entire internet. They, in effect, find similar historical cases to your query and merge them. In many cases, for specific enough queries, the text is verbatim from an original source. This is less revolutionary than the spreadsheet; it's less than google search. It's a speed boost to what was always the most wrote element to what we do. Yes, that often took us the longest -- and so some might be afraid that's what labour is -- but it isnt. We never "added value" to products via what may be automated. Value is always a matter of the desire of the buyer of the products of our labour (vs. the supply) -- and making those products for those buyers was always what they wanted. This will be clear to everyone pretty quickly, as with all tech, it's "magic" on the first encounter -- until the limitations are exposed. I actually work in an area where what took 3mo last year, I can now do in maybe 3 days due to ChatGPT. But when it comes to providing my customers with that content, the value was always in how I provided it and what it did for them. I think this makes my skills more valuable, not less. Since the quality of products will be even more stratified by experts who can quickly assemble what the customer needs from non-experts who have to fight through AI dialogue to get something generic.
protortyp 3y ago
I already use gpt4 for this. Works quite well if you show it a couple of reference unit tests.
toolslive 3y ago
you can quite easily use a GPT to generate test examples. For example, "generate me 1000 customer support mails about ..."

Copilot does a decent job with "dumb" unit tests, e.g. "when the condition X is Y then Z". But it obviously doesn't have knowledge of your domain, and I don't think any LLM could do a perfect job without that.

byby 3y ago

I don't think you get it. The game has already changed. And nobody trained the AI to write unit tests or refactor code. It could do it as emergent behavior. It happened so fast that so many people are in denial and many people aren't even asking the obvious questions. Simply ASK chatGPT to refactor your code to be more unit testable and IT Can do it. Of course it clearly has your typical LLM problems but it is roughly 50% there. The game has changed. If AI gets to 100%, not only will the game be changed, but humans will no longer be part of the game. See what I asked chatGPT: ME: def (x: int): for i in range(x): print(i) Can you refactor the code so it can be more unit testable? CHATGPT: Certainly! To make the code more unit testable, we can refactor it into a function that returns a list of integers instead of directly printing them. Here's the refactored code: python def generate_number_list(x: int) -> List[int]: number_list = [] for i in range(x): number_list.append(i) return number_list In this refactored code, we've created a function called generate_number_list that takes an integer x as input. Instead of printing each number, we use a number_list variable to store the generated numbers by appending them in the loop. Finally, we return the number_list. By returning the list instead of printing the numbers directly, we make the code more testable because we can now write unit tests that assert the correctness of the generated list. If boggles my mind that I see threads of people saying "what if we train LLMs to do this or to do that?" or whatever. Guys, just ask the LLM to do what you want in plain English. It can do it.

mjburgess 3y ago
That isn't the game. The game is designing software to requirements. It's writing literature for a new era. It's creating X for A audience with N vauge unspecified needs -- where X is a complex product made of many parts, involving many people, with shifting and changing problems/solutions/requirements. The game was never writing the stack overflow answer -- that was already written.
- byby 3y ago
  >The game was never writing the stack overflow answer -- that was already written. The problem is this was never a stackoverflow question and there was never an answer for it. Try finding it. The LLM is already playing the game because it came up with that answer which is Fully Correct, Out of Thin Air. Look, clearly the LLM can't play the game as well as a trained adept human, but it's definitely playing the game. >The game is designing software to requirements. It's writing literature for a new era. It's creating X for A audience with N vauge unspecified needs -- where X is a complex product made of many parts, involving many people, with shifting and changing problems/solutions/requirements. It can do all of this. It can talk like you and parrot exactly what your saying and also go into more detail and re-frame your words more eloquently. What you're not getting is that all the things you mentioned the LLM can do in actuality to varying degrees to the point where it is in the "game." and at times it does better than us. Likely, you haven't even tried asking it yet.
  - mjburgess 3y ago
    > Fully Correct, Out of Thin Air I think if you're an expert in an area, this effect is easier to see through. You know where the github repo is, where the library example is, which ebooks there area -- etc. and you're mostly at-ease not using them and just writing the solution yourself. These systems are not "fully correct" and not "out of thin area". They are trained on everything ever digitised, including the entire internet. They, in effect, find similar historical cases to your query and merge them. In many cases, for specific enough queries, the text is verbatim from an original source. This is less revolutionary than the spreadsheet; it's less than google search. It's a speed boost to what was always the most wrote element to what we do. Yes, that often took us the longest -- and so some might be afraid that's what labour is -- but it isnt. We never "added value" to products via what may be automated. Value is always a matter of the desire of the buyer of the products of our labour (vs. the supply) -- and making those products for those buyers was always what they wanted. This will be clear to everyone pretty quickly, as with all tech, it's "magic" on the first encounter -- until the limitations are exposed. I actually work in an area where what took 3mo last year, I can now do in maybe 3 days due to ChatGPT. But when it comes to providing my customers with that content, the value was always in how I provided it and what it did for them. I think this makes my skills more valuable, not less. Since the quality of products will be even more stratified by experts who can quickly assemble what the customer needs from non-experts who have to fight through AI dialogue to get something generic.

protortyp 3y ago

I already use gpt4 for this. Works quite well if you show it a couple of reference unit tests.

toolslive 3y ago

you can quite easily use a GPT to generate test examples. For example, "generate me 1000 customer support mails about ..."