3 ms·
Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it
by ericpauley 5y ago
Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it would be really interesting to see the results. Could be that Copilot is doing no better (or worse) than developers copying from Stack Overflow.
Perhaps you could take a similar approach as [1] and leverage MOOC participants?
[1] https://dl.acm.org/doi/pdf/10.1145/3383773 https://dl.acm.org/doi/pdf/10.1145/3383773
- kiwih 5y agoThanks for your feedback! > My main question when reading is how the results compared to manually-written code. Ah, this is exactly the question. But as you say, much harder to answer. Even if you run a competition, unless you can encourage a wide range of developers to enter, you won't be getting the real value. Instead you might be getting incidence rates of code written by students/interns. Perhaps if you could get a few FAANGs on board to either share internal data (unlikely) or send a random sample of employees (also very unlikely) to make teams and then evaluate their code... It seems like a difficult question to answer. We think a more doable way would be to take snapshots of large open source codebases (e.g. off GitHub) and measure the incidence rate of CWEs, but this also presents its own challenges with analyzing the data. Also, what's the relationship between open source code and all code? Lots of avenues to consider.