5 ms·
Italy did an accidental experiment on the effectiveness of GitHub Copilot
- dadadad100 3y agoBy turning off access to GPTs the Italian government showed how their country’s developers felt about copilot
- jruohonen 3y agoLike with most event studies, they don't control any other events that might possibly affect the likelihoods. It seems that the ban also occurred in a weekend. They should also show the raw time series from a longer period in order to see how stable the series are and thus how realistic the effects are.
- deleted 3y ago[deleted]
- bonzini 3y agoCopilot was not included in the ban, only ChatGPT. Flagging.
- dadadad100 3y agoYep. Sloppy on my part. Suggested better title above but I can’t change it myself.
- koliber 3y agoI wonder if the 50% drop was because developers were half as effective without copilot. Or was it because they have spent half of the day looking for ways to bypass the block while having angry conversations with colleagues, and therefore, were not coding.
- boringuser2 3y agoThe idea that aggregate developer output is 50% improved by the simple existence of transformers seems to not pass my sniff test. I haven't seen any of my coworkers improve their output in the past x years.
- justrealist 3y agoIt's easily true for me. Far more than 50% for someone doing simple React development or DS scripting.
- thefz 3y agoI still think that if your productivity is seriously impacted by chatGPT, you are easily replaceable as a programmer.
- jonplackett 3y agoI’m curious what job you are doing that you don’t feel chatGPT can help with?
- howrar 3y agoTalking about Copilot, not ChatGPT, but it saves so much time when it comes to boilerplate code, naming variables, writing comments/documentation/descriptive error messages. Things that are generally easier to read and verify than to write. The ability to do these things is not where my value lies as a programmer.
- justrealist 3y agoI mean that's kind of silly. Everyone doing "hard" work, whether it's a software engineer developing novel algorithms or a PI at a lab researching open questions spends at least 80% of their time grinding through operational necessities. Writing tests, writing grants, debugging... nobody is sitting a a chalkboard 100% focused on the bleeding edge. GPT turns that 80% into 40%. That's a lot!
- boxed 3y ago> Release event I don't understand this. Did they measure the number of releases in the GitHub "releases" feature?
- leadingthenet 3y agoI was interested to see how it affected them in the medium-term, including the VPN/Tor usage, but the data range seems extremely short, a few days before the ban and a few after. Also doesn’t seem to account for any other events taking place in and around that time, or that it’s a weekend and so on. Doesn’t really say much, imo.
- Filligree 3y agoThe link says nothing about Copilot, this title is not just editorialised but wrong.
- phreeza 3y ago@dang: this post is marked as flagged at the moment, but I don't think it should be? Except maybe for the editorialized title, but that doesn't seem super bad.
- dadadad100 3y agoOP here. Happy to change the title to “on the effectiveness of ChatGPT for GitHub users”. Sloppy reading on my part. Sorry