This is far, far more rigorous than the experiment Microsoft behind Microsoft's claim that Copilot made devs 55% faster. The experiment in question was to spli

This is far, far more rigorous than the experiment Microsoft behind Microsoft's claim that Copilot made devs 55% faster. The experiment in question was to split 95 devs into two groups and see how long it took each group to setup a web server in Javascript. Control took a little under 3 hours on average, the copilot group took 1 hour and 11 minutes on average. https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/#figure-summary-of-the-experiment-process-and-results https://github.blog/news-insights/research/research-quantify... And it is thanks to this weak experiment that Github proudly boasts that Copilot makes devs 55% faster. By contrast the conclusion that Copilot makes devs ~25% more productive seems reasonable, especially when you read the actual paper and find out that among senior devs the productivity gains are more marginal.