3 ms·

The last entity I would trust for trustworthy data on how much Copilot helps programmers is GitHub. They obviously skew the scenarios and metrics to find the be

The last entity I would trust for trustworthy data on how much Copilot helps programmers is GitHub. They obviously skew the scenarios and metrics to find the best possible number to report. I think the truth as revealed in your comment and many others is that it all depends on the context. For repetitive code or boilerplate, or starting new projects in well-known frameworks and languages, it probably does increase velocity. The other context factor is the programmer themselves. Their familiarity with the problem domain, the language, and the framework all matter, as does the personality and coding style of the individual.

claytongulick 2y ago
It also depends on what tools a developer is already using as a productivity booster. For example, I use vanilla web components which have some boilerplate for new components. I already have a simple vscode snippet that does the job well, with no hallucinations [1]. I've experimented with llms doing the same thing with not great results. It took me longer to explain what I wanted than it did for me to just write that snippet. Doing it repeatedly and waiting for the results definitely didn't increase my speed, though I was impressed that it was eventually able to figure it out (vanilla web components with lit-html renderer isn't a super common technique). Also, I prefer the pythonic approach of snake-case local variable names. Getting the llm to do that in a js project where it's not super common was another whole iteration. We've had code generation tools around for decades to deal with repetitive boilerplate tasks. Maybe they aren't quite as capable as when the llm "gets it right" - but I wonder with these productivity claims how they are measured. Are they starting from zero and a new developer? Or comparing against an experienced developer with proficiency with lots of workflow enhancers like templates, snippets, vim macros, etc... [1] https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f31761b0f293 https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f3...

claytongulick 2y ago

It also depends on what tools a developer is already using as a productivity booster. For example, I use vanilla web components which have some boilerplate for new components. I already have a simple vscode snippet that does the job well, with no hallucinations [1]. I've experimented with llms doing the same thing with not great results. It took me longer to explain what I wanted than it did for me to just write that snippet. Doing it repeatedly and waiting for the results definitely didn't increase my speed, though I was impressed that it was eventually able to figure it out (vanilla web components with lit-html renderer isn't a super common technique). Also, I prefer the pythonic approach of snake-case local variable names. Getting the llm to do that in a js project where it's not super common was another whole iteration. We've had code generation tools around for decades to deal with repetitive boilerplate tasks. Maybe they aren't quite as capable as when the llm "gets it right" - but I wonder with these productivity claims how they are measured. Are they starting from zero and a new developer? Or comparing against an experienced developer with proficiency with lots of workflow enhancers like templates, snippets, vim macros, etc... [1] https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f31761b0f293 https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f3...