3 ms·

I use Github co-pilot but 55% productive is a bullshit number. Perhaps 1% may be. Most of co-pilot suggestions are either simple pattern matches or subtle hallu

by nojvek 2y ago

I use Github co-pilot but 55% productive is a bullshit number. Perhaps 1% may be. Most of co-pilot suggestions are either simple pattern matches or subtle hallucinations where I have to catch and fix silly bugs. Github Chat is not very useful at understanding what the code is doing. Tried it once or twice and gave up. The hype will help with Microsoft Stock though. Seems like bean counter management is taking over.

Bjorkbat 2y ago
I was actually about to make a comment on that. They got that number from some dumb "experiment" they did where told two groups of developers to implement an HTTP server in Javascript. The group with Copilot got the job done 55% faster (https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?ocid=AIDcmmc3fhtaow_SEM__k_Cj0KCQjwir2xBhC_ARIsAMTXk85qF--5Z_RO3fCcRnFE6itQ3cKFQvIGLnsyaF-Gwd4FdZFoisM7GtUaAou4EALw_wcB_k_#finding-2-but-speed-is-important-too https://github.blog/2022-09-07-research-quantifying-github-c...) So yeah, they did an experiment with <100 developers asking them to create implement something that only took the control group 3 hours to finish from scratch, and from this we got the "55% more productive" statistic.
- chasd00 2y ago
  That's really disappointing and a little bit scary. That seems like a task tailor fit for an LLM, it's been done a million times before, follows an established protocol, and the implementations are all basically the same. Granted a lot of code is like that but a more realistic task may have been something like "create an online ordering system for a cake company that sells different kinds of cakes, has a sale every so often, and uses coupons printed in the local paper". That task is more ambiguous and better reflects what software devs are tasked with day to day IMO.
- deleted 2y ago
  [deleted]

by nojvek 2y ago

Bjorkbat 2y ago

I was actually about to make a comment on that. They got that number from some dumb "experiment" they did where told two groups of developers to implement an HTTP server in Javascript. The group with Copilot got the job done 55% faster (https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?ocid=AIDcmmc3fhtaow_SEM__k_Cj0KCQjwir2xBhC_ARIsAMTXk85qF--5Z_RO3fCcRnFE6itQ3cKFQvIGLnsyaF-Gwd4FdZFoisM7GtUaAou4EALw_wcB_k_#finding-2-but-speed-is-important-too https://github.blog/2022-09-07-research-quantifying-github-c...) So yeah, they did an experiment with <100 developers asking them to create implement something that only took the control group 3 hours to finish from scratch, and from this we got the "55% more productive" statistic.

chasd00 2y ago
That's really disappointing and a little bit scary. That seems like a task tailor fit for an LLM, it's been done a million times before, follows an established protocol, and the implementations are all basically the same. Granted a lot of code is like that but a more realistic task may have been something like "create an online ordering system for a cake company that sells different kinds of cakes, has a sale every so often, and uses coupons printed in the local paper". That task is more ambiguous and better reflects what software devs are tasked with day to day IMO.
deleted 2y ago
[deleted]