7 ms·

I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60

I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))

aliencat 4y ago
unit test definitely a great use of copilot.
- wokwokwok 4y ago
  What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes). Other people have said this, but copilot is auto complete. You use it every time you press tab. Do you accept an auto-complete suggestion 40% of the time? ...mmmm, yes, well, guess why 40% of code is generated by copilot. It's not because people are generating tests with it.
  - rjh29 4y ago
    Most of the time I know exactly what tests I want to write but it's annoying to write them. Copilot autocompletes, I check it and then I use it.
    - Waterluvian 4y ago
      For me the worst part about this is that writing tests tediously makes me reach for layers of abstraction on my test code. And then suddenly my test code is complicated and needs its own test code, and changing a test can often be problematic to the abstractions I foolishly employed. Being able to churn out the boilerplate for tests, which can make DRY a non-concern, is great. I just hope that if I do this, I never get sloppy, and I always review the tests.
      - rjh29 4y ago
        I agree! I try not to make test code too complicated, and prefer repetitive code over abstractions (else you have to test the tests!) so Copilot is useful there.
  - dandellion 4y ago
    In my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.
    - paro_nej 4y ago
      > In my case about 50% of the test code is just boilerplate Why not replace it with a function call?
      - aaomidi 4y ago
        Some languages are just noisy. Go, Java I’ve experienced are noisier than others.
      - travisjungroth 4y ago
        A common occurrence is something like a pool of 10 actions where a bunch of tests each do 3 to 7 of them. This is very hard to abstract with a function call.
      - lupire 4y ago
        That makes the test harder to read. Tests should be dumb.
      - dandellion 4y ago
        In the case I was referring to, mainly to keep the code consistent with what was already there and the PRs small, as the code base is primarily owned by a different team. It's also pretty innocuous short tests that read well as they are.
  - teaearlgraycold 4y ago
    I use it for unit tests. That's one of its best use cases. Obviously I read the code it writes down.
  - kakwa_ 4y ago
    In a recent personal C project (a custom archive file extractor), I have used chatgpt to generate tons of unit tests. And honestly, I was extremely impressed. With a bit of context, it was able to generate almost correct test data for a fairly complex binary data format. I've also leverage it heavily to: * generate doxygen comments * get usage examples of libs I never used * create a whole bunch of utils functions. And honestly for all these tedious tasks, it has done a far better job than I would ever had. Comments were consistent in styling, the base examples were fairly good, and the utils functions were well made, specially the error handling (which I would probably have semi-consciously skipped tbh). In fairness, I modified most of this code slightly to make it fit my project structure, or tweaked it a bit what the IA didn't quite get it right. (it never fully understood some offset fields in the file format, but got pretty close to at times. And in fairness my naming for these offset fields was a bit questionable). Heck, as a test, I even threw at it an RFC-like spec of the file format, and asked him to generate a python parser. The result was not 100% correct, but definitely a good start to iterate on. In the end, this side project took me 2 weeks to implement with chatgpt probably saving around 1 week of dev. It also greatly helped improving the quality of the project (better doc, better tests).
  - bananapub 4y ago
    > I don't know anyone who uses it for this that would be surprising - it does a fantastic job of producing e.g. dumb unit tests. for instance, Copilot + a Go table testing template means you can churn out the simple "make a request to this url with this data, ensure I get a 200 and the response contains 'id' and not 'error'" extremely quickly. the code is trivial but tedious, so you can quickly inspect for sanity and run to ensure they pass, then commit and have them checking future changes. > it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes). variations on this comment are all over these threads, which is bizarre. hallucinating means you waste a few seconds reading the code it produced, not that you commit incorrect code. this isn't like ChatGPT advising people to drink bleach, Copilot is a dumb tool offering an expert (you) suggested solutions for your expert consideration
- wongarsu 4y ago
  I guess this could be true in some languages and settings, for example when a notable portion of your tests verify correct behavior with null values, arguments of wrong type, etc. When talking about unit tests that verify the actual logic, you should naturally become more careful with your copilot suggestions. Especially if copilot also wrote the code it's testing.
cortesoft 4y ago
Yeah, to me this doesn’t feel that different than using a library from another developer, just slightly more customized. Most code that is run is not written by the final developer, but by people further up the chain.
js8 4y ago
The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.
- layer8 4y ago
  I agree to some extent. But there are redundancies that are inherently beneficial and make code more robust and readable, like separating interface from implementation (where some declarative elements have to be repeated), or explicit type annotations instead of full-program type inference. There are also strictly necessary redundancies, like when you pattern-match an ADT, you repeat its declared structure. For desirable or necessary redundancies like that, AI code completion can still save time. Intelligent unit test creation (if it is indeed intelligent) is also a good use-case. Not to relieve the programmer from thinking, but to save typing and to point out potentially useful tests that you might not have thought of.
  - nine_k 4y ago
    This is trivial stuff which is handled well by any competent IDE from 2015, and even a well enough-working set of editor templates. I don't see where "AI" is required for stuff like this.
    - js8 4y ago
      I agree, but also, this is actually where AI (whatever it is) would be helpful. Suggesting actual abstractions that could replace boilerplate. Of course, these can build on existing abstractions.
- lupire 4y ago
  The boilerplate makes it readable by humans.
- never_inline 4y ago

aliencat 4y ago

unit test definitely a great use of copilot.

wokwokwok 4y ago
What are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes). Other people have said this, but copilot is auto complete. You use it every time you press tab. Do you accept an auto-complete suggestion 40% of the time? ...mmmm, yes, well, guess why 40% of code is generated by copilot. It's not because people are generating tests with it.
- rjh29 4y ago
  Most of the time I know exactly what tests I want to write but it's annoying to write them. Copilot autocompletes, I check it and then I use it.
  - Waterluvian 4y ago
    For me the worst part about this is that writing tests tediously makes me reach for layers of abstraction on my test code. And then suddenly my test code is complicated and needs its own test code, and changing a test can often be problematic to the abstractions I foolishly employed. Being able to churn out the boilerplate for tests, which can make DRY a non-concern, is great. I just hope that if I do this, I never get sloppy, and I always review the tests.
    - rjh29 4y ago
      I agree! I try not to make test code too complicated, and prefer repetitive code over abstractions (else you have to test the tests!) so Copilot is useful there.
- dandellion 4y ago
  In my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.
  - paro_nej 4y ago
    > In my case about 50% of the test code is just boilerplate Why not replace it with a function call?
    - aaomidi 4y ago
      Some languages are just noisy. Go, Java I’ve experienced are noisier than others.
    - travisjungroth 4y ago
      A common occurrence is something like a pool of 10 actions where a bunch of tests each do 3 to 7 of them. This is very hard to abstract with a function call.
    - lupire 4y ago
      That makes the test harder to read. Tests should be dumb.
    - dandellion 4y ago
      In the case I was referring to, mainly to keep the code consistent with what was already there and the PRs small, as the code base is primarily owned by a different team. It's also pretty innocuous short tests that read well as they are.
- teaearlgraycold 4y ago
  I use it for unit tests. That's one of its best use cases. Obviously I read the code it writes down.
- kakwa_ 4y ago
  In a recent personal C project (a custom archive file extractor), I have used chatgpt to generate tons of unit tests. And honestly, I was extremely impressed. With a bit of context, it was able to generate almost correct test data for a fairly complex binary data format. I've also leverage it heavily to: * generate doxygen comments * get usage examples of libs I never used * create a whole bunch of utils functions. And honestly for all these tedious tasks, it has done a far better job than I would ever had. Comments were consistent in styling, the base examples were fairly good, and the utils functions were well made, specially the error handling (which I would probably have semi-consciously skipped tbh). In fairness, I modified most of this code slightly to make it fit my project structure, or tweaked it a bit what the IA didn't quite get it right. (it never fully understood some offset fields in the file format, but got pretty close to at times. And in fairness my naming for these offset fields was a bit questionable). Heck, as a test, I even threw at it an RFC-like spec of the file format, and asked him to generate a python parser. The result was not 100% correct, but definitely a good start to iterate on. In the end, this side project took me 2 weeks to implement with chatgpt probably saving around 1 week of dev. It also greatly helped improving the quality of the project (better doc, better tests).
- bananapub 4y ago
  > I don't know anyone who uses it for this that would be surprising - it does a fantastic job of producing e.g. dumb unit tests. for instance, Copilot + a Go table testing template means you can churn out the simple "make a request to this url with this data, ensure I get a 200 and the response contains 'id' and not 'error'" extremely quickly. the code is trivial but tedious, so you can quickly inspect for sanity and run to ensure they pass, then commit and have them checking future changes. > it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes). variations on this comment are all over these threads, which is bizarre. hallucinating means you waste a few seconds reading the code it produced, not that you commit incorrect code. this isn't like ChatGPT advising people to drink bleach, Copilot is a dumb tool offering an expert (you) suggested solutions for your expert consideration
wongarsu 4y ago
I guess this could be true in some languages and settings, for example when a notable portion of your tests verify correct behavior with null values, arguments of wrong type, etc. When talking about unit tests that verify the actual logic, you should naturally become more careful with your copilot suggestions. Especially if copilot also wrote the code it's testing.

cortesoft 4y ago

Yeah, to me this doesn’t feel that different than using a library from another developer, just slightly more customized. Most code that is run is not written by the final developer, but by people further up the chain.

js8 4y ago

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

layer8 4y ago
I agree to some extent. But there are redundancies that are inherently beneficial and make code more robust and readable, like separating interface from implementation (where some declarative elements have to be repeated), or explicit type annotations instead of full-program type inference. There are also strictly necessary redundancies, like when you pattern-match an ADT, you repeat its declared structure. For desirable or necessary redundancies like that, AI code completion can still save time. Intelligent unit test creation (if it is indeed intelligent) is also a good use-case. Not to relieve the programmer from thinking, but to save typing and to point out potentially useful tests that you might not have thought of.
- nine_k 4y ago
  This is trivial stuff which is handled well by any competent IDE from 2015, and even a well enough-working set of editor templates. I don't see where "AI" is required for stuff like this.
  - js8 4y ago
    I agree, but also, this is actually where AI (whatever it is) would be helpful. Suggesting actual abstractions that could replace boilerplate. Of course, these can build on existing abstractions.
lupire 4y ago
The boilerplate makes it readable by humans.
never_inline 4y ago