4 ms·

> I continue to be surprised GitHub shows examples that wouldn't compile/run correctly. Why is that surprising to you? CoPolit doesn't actually know how to cod

by qayxc 5y ago

> I continue to be surprised GitHub shows examples that wouldn't compile/run correctly. Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Sometimes these generated symbols don't represent valid code and since CoPilot doesn't actually perform filtering based on syntax checks or JIT, these results end up as suggestions. This is actually a point where future versions could greatly improve the usefulness, e.g. use the compiler infrastructure to verify and filter generated results. This includes auto-formatting and even result scoring by code metrics (conciseness, complexity, ...). Plenty of room for improvement even without touching the underlying model.

iudqnolq 5y ago
I'm not surprised the actual results are flawed. I'm surprised the handful of hand-picked examples Github calls out are flawed. Usually handpicked examples of algorithms are the best-case performance.
- pjot 5y ago
  Maybe it’s just a more honest representation of a typical result? Looking at it the other way around, had they chose _perfect_ examples there’d be similar criticism about “cherry picking”.
  - iudqnolq 5y ago
    Fair enough, although it makes me think that if these are the better examples the average is much worse.
quicklime 5y ago
There was an incorrect example on the copilot.github.com landing page which I was surprised by. I'm guessing someone noticed it because it's been taken off the page. It was in a file called "find_files.sh", and the command it generated was something like: > find .... -exec ... {}\; The problem is that there's no whitespace between the {} and \; and this is rejected by both GNU and BSD find. It might've just been a mistake that someone made while putting the landing page together. But if it was generated, then how?? I can't imagine there are many (if any) instances of "{}\;" in the training data, since it's not a valid invocation of the find command...
- tyingq 5y ago
  Here it is, courtesy of a wayback snapshot[1] #!/bin/bash # List all python source files which are more than 1KB and contain the word "copilot". find . \ -name "\*.py" \ -size +1000 \ -exec grep -n copilot {}\; [1] https://web.archive.org/web/20210701061847if_/https://copilot.github.com/ https://web.archive.org/web/20210701061847if_/https://copilo...
  - onionisafruit 5y ago
    The missing space isn't the only thing wrong there. - This lists the lines containing copilot instead of the files. You want `grep -l` to list the files. - `--size +1000` finds files with over 1000 blocks. `--size +1000c` gets you files over 1KB. - `-name "*.py"` only finds files literally named `.py`. To find files that match the glob `.py`, you would single quote it like `-name '*.py'`
hliyan 5y ago
> Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Idea: make Copilot test-driven. You provide a function invocation and an example output, and it searches for code with similar test cases, then mutate the code until the test case passes within its own runtime (it may need to mock certain things to make tests pass within its own runtime).

by qayxc 5y ago

iudqnolq 5y ago

I'm not surprised the actual results are flawed. I'm surprised the handful of hand-picked examples Github calls out are flawed. Usually handpicked examples of algorithms are the best-case performance.

pjot 5y ago
Maybe it’s just a more honest representation of a typical result? Looking at it the other way around, had they chose _perfect_ examples there’d be similar criticism about “cherry picking”.
- iudqnolq 5y ago
  Fair enough, although it makes me think that if these are the better examples the average is much worse.

quicklime 5y ago

There was an incorrect example on the copilot.github.com landing page which I was surprised by. I'm guessing someone noticed it because it's been taken off the page. It was in a file called "find_files.sh", and the command it generated was something like: > find .... -exec ... {}\; The problem is that there's no whitespace between the {} and \; and this is rejected by both GNU and BSD find. It might've just been a mistake that someone made while putting the landing page together. But if it was generated, then how?? I can't imagine there are many (if any) instances of "{}\;" in the training data, since it's not a valid invocation of the find command...

tyingq 5y ago
Here it is, courtesy of a wayback snapshot[1] #!/bin/bash # List all python source files which are more than 1KB and contain the word "copilot". find . \ -name "\*.py" \ -size +1000 \ -exec grep -n copilot {}\; [1] https://web.archive.org/web/20210701061847if_/https://copilot.github.com/ https://web.archive.org/web/20210701061847if_/https://copilo...
- onionisafruit 5y ago
  The missing space isn't the only thing wrong there. - This lists the lines containing copilot instead of the files. You want `grep -l` to list the files. - `--size +1000` finds files with over 1000 blocks. `--size +1000c` gets you files over 1KB. - `-name "*.py"` only finds files literally named `.py`. To find files that match the glob `.py`, you would single quote it like `-name '*.py'`

hliyan 5y ago

> Why is that surprising to you? CoPolit doesn't actually know how to code, it just generates symbols that match learned patterns. Idea: make Copilot test-driven. You provide a function invocation and an example output, and it searches for code with similar test cases, then mutate the code until the test case passes within its own runtime (it may need to mock certain things to make tests pass within its own runtime).