6 ms·

While somewhat off-topic, I had an interesting experience highlighting the utility of GitHub's Copilot the other day. I decided to run Copilot on a piece of cod

by _pdp_ 3y ago

While somewhat off-topic, I had an interesting experience highlighting the utility of GitHub's Copilot the other day. I decided to run Copilot on a piece of code functioning correctly to see if it would identify any non-existent issues. Surprisingly, it managed to pinpoint an actual bug. Following this discovery, I asked Copilot to generate a unit test to better understand the identified issue. Upon running the test, the program crashed just as Copilot had predicted. I then refactored the problematic lines as per Copilot's suggestions. This was my first time witnessing the effectiveness of Copilot in such a scenario, which provided small yet significant proof to me that Language Models can be invaluable tools for coding, capable of identifying and helping to resolve real bugs. Although they may have limitations, I believe any imperfections are merely temporary hurdles toward more robust coding assistants.

WanderPanda 3y ago
Copilot at present capabilities is already so valuable that not having it in some environment gives me the „disabledness feeling“ that I otherwise only get when vim bindings are not enabled. Absolute miracle technology! I‘m sure in the not too distant future we‘ll have have privacy preserving, open source versions that are good enough to not shovel everything over to openai
- jgalt212 3y ago
  > shovel everything over to openai Seriously, if you're a niche market with specific know-how, the easiest way to broadly propagate this know-now is to use copilot.
ushakov 3y ago
That’s rather an exception in my experience. For unit-tests it starts hallucinating hard once you have functions imported from other files. This is probably the reason most unit tests in their marketing materials are things like fibonacci…
chrisco255 3y ago
Try it on a million line code base where it's not so cut and dry to even determine if the code is running correctly or what correctly means when it changes day to day.
- ushakov 3y ago
  You can’t, it has a context size window of 8192 tokens. That’s like 1000 lines depending on programming language
- yjftsjthsd-h 3y ago
  Is it better or worse than a human, though?
  - SirMaster 3y ago
    I would imagine worse, because a human has a much, much, much larger context size.
    - jameshart 3y ago
      But also a much much shorter attention span and tolerance for BS. If you ask the LLM to analyze those 1000000 lines 1000 at a time, 1000 times, it’ll do it, with the same diligence and attention to detail across all 1000 pages. Ask a human to do it and their patience will be tested. Their focus will waver, they’ll grow used to patterns and miss anomalies, and they’ll probably skip chunks that look fine at first glance. Sure the LLM won’t find big picture issues at that scale. But it’ll find plenty of code smells and minor logic errors that deserve a second look.
      - chrisco255 3y ago
        Ok, why don't you run this experiment on a large public open source code base? We should be drowning in valuable bug reports right now but all I hear is hype.
    - Cthulhu_ 3y ago
sangnoir 3y ago
That sounds like very basic code review - which I guess is useful in instances where one can't get a review from a human. If it has a low enough false-positive rate, it could be great as a background CI/CD bot that can chime in the PR/changeset comments to say "You may have a bug here"
- TheBlight 3y ago
  One nice thing about a machine code reviewing is no tedious passive-aggressive interactions or subjective style feedback you feel compelled to take etc.
  - crazysim 3y ago
    This isn't always the case! There was a code review Q/A model posted on /r/locallamas which was very amusingly StackOverflow sometimes.
- ceedan 3y ago
  Discovering a bug, and reproducing via unit tests is very different than "a very basic code review"
  - sangnoir 3y ago
    Identifying potential bugs within a unit is only a part of a good code review; good code reviews also identify potential issues with broader system goals, readability, and idiomaticness, elegance and "taste" (e.g. pythonicity in Python) which require larger contexts than LLMs can currently muster. So yes, the ability to identify a bug and providing a unit test to reproduce it is rather basic[1], compared to what a good code review can be. 1. An org I worked for had one such question for entry-level SWEs interviews in 3-parts: What's wrong with this code? Design test cases for it. Write the correct version (and check if the tests pass)
    - deleted 3y ago
      [deleted]
rripken 3y ago
How did you prompt Copilot to identify issues? In my experience the best I can do is to put in code comments of what what I want a snippet to do and copilot tries to write it. I haven't had good luck asking copilot to rewrite existing code. Nearest I've gotten is: // method2 is identical to method1 except it fixes the bugs public void method2(){
- crazysim 3y ago
  Might be using the Copilot chat feature.
ChatGTP 3y ago
These things are amazing when you first experience but I think in most cases the user fails to realise how common their particular bug is. But then you also need to realise there maybe bugs in what has been suggested. We all know there are issue with stack overflow responses too. Probably 85% of codebase are just rehashes of the same stuff. Co-pilot has seen it all I guess.
pylua 3y ago
This is a great use of ai. In all seriousness I can’t wait for the day it gets added to spring as a plug-in.

by _pdp_ 3y ago

WanderPanda 3y ago

Copilot at present capabilities is already so valuable that not having it in some environment gives me the „disabledness feeling“ that I otherwise only get when vim bindings are not enabled. Absolute miracle technology! I‘m sure in the not too distant future we‘ll have have privacy preserving, open source versions that are good enough to not shovel everything over to openai

jgalt212 3y ago
> shovel everything over to openai Seriously, if you're a niche market with specific know-how, the easiest way to broadly propagate this know-now is to use copilot.

ushakov 3y ago

That’s rather an exception in my experience. For unit-tests it starts hallucinating hard once you have functions imported from other files. This is probably the reason most unit tests in their marketing materials are things like fibonacci…

chrisco255 3y ago

Try it on a million line code base where it's not so cut and dry to even determine if the code is running correctly or what correctly means when it changes day to day.

ushakov 3y ago
You can’t, it has a context size window of 8192 tokens. That’s like 1000 lines depending on programming language
yjftsjthsd-h 3y ago
Is it better or worse than a human, though?
- SirMaster 3y ago
  I would imagine worse, because a human has a much, much, much larger context size.
  - jameshart 3y ago
    But also a much much shorter attention span and tolerance for BS. If you ask the LLM to analyze those 1000000 lines 1000 at a time, 1000 times, it’ll do it, with the same diligence and attention to detail across all 1000 pages. Ask a human to do it and their patience will be tested. Their focus will waver, they’ll grow used to patterns and miss anomalies, and they’ll probably skip chunks that look fine at first glance. Sure the LLM won’t find big picture issues at that scale. But it’ll find plenty of code smells and minor logic errors that deserve a second look.
    - chrisco255 3y ago
      Ok, why don't you run this experiment on a large public open source code base? We should be drowning in valuable bug reports right now but all I hear is hype.
  - Cthulhu_ 3y ago

sangnoir 3y ago

That sounds like very basic code review - which I guess is useful in instances where one can't get a review from a human. If it has a low enough false-positive rate, it could be great as a background CI/CD bot that can chime in the PR/changeset comments to say "You may have a bug here"

TheBlight 3y ago
One nice thing about a machine code reviewing is no tedious passive-aggressive interactions or subjective style feedback you feel compelled to take etc.
- crazysim 3y ago
  This isn't always the case! There was a code review Q/A model posted on /r/locallamas which was very amusingly StackOverflow sometimes.
ceedan 3y ago
Discovering a bug, and reproducing via unit tests is very different than "a very basic code review"
- sangnoir 3y ago
  Identifying potential bugs within a unit is only a part of a good code review; good code reviews also identify potential issues with broader system goals, readability, and idiomaticness, elegance and "taste" (e.g. pythonicity in Python) which require larger contexts than LLMs can currently muster. So yes, the ability to identify a bug and providing a unit test to reproduce it is rather basic[1], compared to what a good code review can be. 1. An org I worked for had one such question for entry-level SWEs interviews in 3-parts: What's wrong with this code? Design test cases for it. Write the correct version (and check if the tests pass)
  - deleted 3y ago
    [deleted]

rripken 3y ago

How did you prompt Copilot to identify issues? In my experience the best I can do is to put in code comments of what what I want a snippet to do and copilot tries to write it. I haven't had good luck asking copilot to rewrite existing code. Nearest I've gotten is: // method2 is identical to method1 except it fixes the bugs public void method2(){

crazysim 3y ago
Might be using the Copilot chat feature.

ChatGTP 3y ago

These things are amazing when you first experience but I think in most cases the user fails to realise how common their particular bug is. But then you also need to realise there maybe bugs in what has been suggested. We all know there are issue with stack overflow responses too. Probably 85% of codebase are just rehashes of the same stuff. Co-pilot has seen it all I guess.

pylua 3y ago

This is a great use of ai. In all seriousness I can’t wait for the day it gets added to spring as a plug-in.