4 ms·

I continue to be surprised GitHub shows examples that wouldn't compile/run correctly. For example, the Wikipedia scraping example that the author claims is also

I continue to be surprised GitHub shows examples that wouldn't compile/run correctly. For example, the Wikipedia scraping example that the author claims is also the intuitive way to solve the problem assigns each row to the global variable cols instead of appending. Further, the following if statement appears to be mis-indented.

qayxc 5y ago
> I continue to be surprised GitHub shows examples that wouldn't compile/run correctly. Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Sometimes these generated symbols don't represent valid code and since CoPilot doesn't actually perform filtering based on syntax checks or JIT, these results end up as suggestions. This is actually a point where future versions could greatly improve the usefulness, e.g. use the compiler infrastructure to verify and filter generated results. This includes auto-formatting and even result scoring by code metrics (conciseness, complexity, ...). Plenty of room for improvement even without touching the underlying model.
- iudqnolq 5y ago
  I'm not surprised the actual results are flawed. I'm surprised the handful of hand-picked examples Github calls out are flawed. Usually handpicked examples of algorithms are the best-case performance.
  - pjot 5y ago
    Maybe it’s just a more honest representation of a typical result? Looking at it the other way around, had they chose _perfect_ examples there’d be similar criticism about “cherry picking”.
    - iudqnolq 5y ago
      Fair enough, although it makes me think that if these are the better examples the average is much worse.
- quicklime 5y ago
  There was an incorrect example on the copilot.github.com landing page which I was surprised by. I'm guessing someone noticed it because it's been taken off the page. It was in a file called "find_files.sh", and the command it generated was something like: > find .... -exec ... {}\; The problem is that there's no whitespace between the {} and \; and this is rejected by both GNU and BSD find. It might've just been a mistake that someone made while putting the landing page together. But if it was generated, then how?? I can't imagine there are many (if any) instances of "{}\;" in the training data, since it's not a valid invocation of the find command...
  - tyingq 5y ago
    Here it is, courtesy of a wayback snapshot[1] #!/bin/bash # List all python source files which are more than 1KB and contain the word "copilot". find . \ -name "\*.py" \ -size +1000 \ -exec grep -n copilot {}\; [1] https://web.archive.org/web/20210701061847if_/https://copilot.github.com/ https://web.archive.org/web/20210701061847if_/https://copilo...
    - onionisafruit 5y ago
      The missing space isn't the only thing wrong there. - This lists the lines containing copilot instead of the files. You want `grep -l` to list the files. - `--size +1000` finds files with over 1000 blocks. `--size +1000c` gets you files over 1KB. - `-name "*.py"` only finds files literally named `.py`. To find files that match the glob `.py`, you would single quote it like `-name '*.py'`
- hliyan 5y ago
  > Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Idea: make Copilot test-driven. You provide a function invocation and an example output, and it searches for code with similar test cases, then mutate the code until the test case passes within its own runtime (it may need to mock certain things to make tests pass within its own runtime).
rvz 5y ago
It's hardly surprising to see a so-called 'AI system' (If you can even call it that) suggesting and generating broken code. A single typo is enough to confuse it and give you the wrong implementation that won't compile. [0] The examples all look cherry picked for presentation; but the ones who had access to Copilot early didn't even stress test it, yet it still generated garbage. And it took 1 in 10 tries to get it right. [1] Of course, we'll hide behind the 'Technical Preview' argument but since the core of this 'AI' (GPT-3) is a black box system, we'll probably never know why it is generating the code it is generating. [0] https://twitter.com/leifg/status/1411083360756146177 https://twitter.com/leifg/status/1411083360756146177 [1] https://news.ycombinator.com/item?id=27676845 https://news.ycombinator.com/item?id=27676845

qayxc 5y ago

> I continue to be surprised GitHub shows examples that wouldn't compile/run correctly. Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Sometimes these generated symbols don't represent valid code and since CoPilot doesn't actually perform filtering based on syntax checks or JIT, these results end up as suggestions. This is actually a point where future versions could greatly improve the usefulness, e.g. use the compiler infrastructure to verify and filter generated results. This includes auto-formatting and even result scoring by code metrics (conciseness, complexity, ...). Plenty of room for improvement even without touching the underlying model.

iudqnolq 5y ago
I'm not surprised the actual results are flawed. I'm surprised the handful of hand-picked examples Github calls out are flawed. Usually handpicked examples of algorithms are the best-case performance.
- pjot 5y ago
  Maybe it’s just a more honest representation of a typical result? Looking at it the other way around, had they chose _perfect_ examples there’d be similar criticism about “cherry picking”.
  - iudqnolq 5y ago
    Fair enough, although it makes me think that if these are the better examples the average is much worse.
quicklime 5y ago
There was an incorrect example on the copilot.github.com landing page which I was surprised by. I'm guessing someone noticed it because it's been taken off the page. It was in a file called "find_files.sh", and the command it generated was something like: > find .... -exec ... {}\; The problem is that there's no whitespace between the {} and \; and this is rejected by both GNU and BSD find. It might've just been a mistake that someone made while putting the landing page together. But if it was generated, then how?? I can't imagine there are many (if any) instances of "{}\;" in the training data, since it's not a valid invocation of the find command...
- tyingq 5y ago
  Here it is, courtesy of a wayback snapshot[1] #!/bin/bash # List all python source files which are more than 1KB and contain the word "copilot". find . \ -name "\*.py" \ -size +1000 \ -exec grep -n copilot {}\; [1] https://web.archive.org/web/20210701061847if_/https://copilot.github.com/ https://web.archive.org/web/20210701061847if_/https://copilo...
  - onionisafruit 5y ago
    The missing space isn't the only thing wrong there. - This lists the lines containing copilot instead of the files. You want `grep -l` to list the files. - `--size +1000` finds files with over 1000 blocks. `--size +1000c` gets you files over 1KB. - `-name "*.py"` only finds files literally named `.py`. To find files that match the glob `.py`, you would single quote it like `-name '*.py'`
hliyan 5y ago
> Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Idea: make Copilot test-driven. You provide a function invocation and an example output, and it searches for code with similar test cases, then mutate the code until the test case passes within its own runtime (it may need to mock certain things to make tests pass within its own runtime).

rvz 5y ago

It's hardly surprising to see a so-called 'AI system' (If you can even call it that) suggesting and generating broken code. A single typo is enough to confuse it and give you the wrong implementation that won't compile. [0] The examples all look cherry picked for presentation; but the ones who had access to Copilot early didn't even stress test it, yet it still generated garbage. And it took 1 in 10 tries to get it right. [1] Of course, we'll hide behind the 'Technical Preview' argument but since the core of this 'AI' (GPT-3) is a black box system, we'll probably never know why it is generating the code it is generating. [0] https://twitter.com/leifg/status/1411083360756146177 https://twitter.com/leifg/status/1411083360756146177 [1] https://news.ycombinator.com/item?id=27676845 https://news.ycombinator.com/item?id=27676845