4 ms·

Setting aside the academic implications for a moment, if you think that Copilot-like models won't be a significant part of programming in the future I'd like to

by ctoth 4y ago

Setting aside the academic implications for a moment, if you think that Copilot-like models won't be a significant part of programming in the future I'd like to point out this research from Google: > We compare the hybrid semantic ML code completion of 10k+ Googlers (over three months across eight programming languages) to a control group and see a 6% reduction in coding iteration time (time between builds and tests) and a 7% reduction in context switches (i.e., leaving the IDE) when exposed to single-line ML completion. These results demonstrate that the combination of ML and SEs can improve developer productivity. Currently, 3% of new code (measured in characters) is now generated from accepting ML completion suggestions. So 3% of all code at Google is now written by AI autocomplete, and Google developers who use this technology are 6% more efficient than those who don't[0]. Is it bad that students are cheating on homework with this? As bad as it was when students used Stackoverflow to cheat, I reckon. But will working with AI models be an essential skill to learn for new (and existing) programmers? About as much as getting comfortable with Stackoverflow was, I reckon. [0]: https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html https://ai.googleblog.com/2022/07/ml-enhanced-code-completio...

beej71 4y ago
Sign of the times, I think. How long until AI offers complete solutions to basic business websites? How long until it does even more?
Test0129 4y ago
I don't mean to be pedantic here but 6% improvement in productivity is a rounding error. You can achieve a higher level of productivity by just removing the ping pong table from the break room. This would indicate to me there is a non-significant improvement in productivity. That seems consistent with my having talked to several senior+ engineers who tried copilot and found it simply wrong or unnecessary. There might be AI written code in the future and surely the "no code" shills will latch onto it with religious fervor. But, much like "no code will eat the world", this too will fail. Until general intelligence is capable of producing large scale systems that need to be coddled and orchestrated, bespoke perhaps even brand new approaches to business problems, etc, there will always be many highly paid humans there. The conspiracy theorist in me makes me think that all of this push to shill this product constantly is yet another way the FAANGS are attempting to dilute the pool enough to lower wages. Engineering is highly skilled, highly technical work. The actual work of someone higher than junior level typically evolved way beyond simple CRUD coding exercises. When you spend more time making sure copilot meets standards, the code is correct, etc it's simply not that useful. It feels much more like the fancy test case generators for some IDEs. For very trivial, borderline rudimentary cases, it works wonderfully and saves time. Once things become complicated and messy it just gets in the way.
- morelisp 4y ago
  It's not even a 6% productivity increase, it's "6% less time between builds and tests". Is that even a meaningful metric? It is also not clear to me what the control group was, neither how they were selected nor what they did instead of using the ML model.

by ctoth 4y ago

beej71 4y ago

Sign of the times, I think. How long until AI offers complete solutions to basic business websites? How long until it does even more?

Test0129 4y ago

I don't mean to be pedantic here but 6% improvement in productivity is a rounding error. You can achieve a higher level of productivity by just removing the ping pong table from the break room. This would indicate to me there is a non-significant improvement in productivity. That seems consistent with my having talked to several senior+ engineers who tried copilot and found it simply wrong or unnecessary. There might be AI written code in the future and surely the "no code" shills will latch onto it with religious fervor. But, much like "no code will eat the world", this too will fail. Until general intelligence is capable of producing large scale systems that need to be coddled and orchestrated, bespoke perhaps even brand new approaches to business problems, etc, there will always be many highly paid humans there. The conspiracy theorist in me makes me think that all of this push to shill this product constantly is yet another way the FAANGS are attempting to dilute the pool enough to lower wages. Engineering is highly skilled, highly technical work. The actual work of someone higher than junior level typically evolved way beyond simple CRUD coding exercises. When you spend more time making sure copilot meets standards, the code is correct, etc it's simply not that useful. It feels much more like the fancy test case generators for some IDEs. For very trivial, borderline rudimentary cases, it works wonderfully and saves time. Once things become complicated and messy it just gets in the way.

morelisp 4y ago
It's not even a 6% productivity increase, it's "6% less time between builds and tests". Is that even a meaningful metric? It is also not clear to me what the control group was, neither how they were selected nor what they did instead of using the ML model.