3 ms·
The methodology seems to be: compare commit activity from 2023 to prior years, without any idea of how many involve Copilot. Then interpret those changes with a
by dweinus 3y ago
The methodology seems to be: compare commit activity from 2023 to prior years, without any idea of how many involve Copilot. Then interpret those changes with assumptions. That seems a bit shakey.
Also: "The projections for 2024 utilize OpenAI's gpt-4-1106-preview Assistant to run a
quadratic regression on existing data." ...am I to understand they asked gpt to do a regression on the data (4 numbers) rather than running a simple regression tool (sklearn, r, even excel can do this)? Even if done correctly, it is not very compelling when based off of 4 data points and accounting for my first concern.
- zemo 3y agocheck out the paper, not just the summary. They explain their methodology. The output has four data points because it’s a summary. The input is … more data than that.
- lolinder 3y agoMore data, but OP is right on the weaknesses of the study—the author posted here [0] and acknowledged that they can't actually say anything about causality, just that 2023 looked different than 2020. [0] https://news.ycombinator.com/item?id=39168841 https://news.ycombinator.com/item?id=39168841
- TOMDM 3y agoI'd bet 2017 looked different from 2020 too.
- dweinus 3y agoI did, that's where my quote is from. The appendix confirms; they ran the regressions on just two inputs: 2022 and 2023 totals.
- gcr 3y agoYou can even see a truncated RankWarning in the output in the appendix. Low rank indeed! Numpy is crying out in agony and the authors didn’t notice They’re using GPT….to write code that does “quadratic regression” on two whole data points…which just extends the slope of each [2022, 2023] line one year further!
- zeroonetwothree 3y agoI’m sympathetic to the study results since I have seen similar things anecdotally but I agree their data is not really warranting the conclusions they reach. For all we know it could because of the covid hiring spree and subsequent layoffs.
- panaetius 3y agoNot even that, the prompt used is "Looking only at the years 2022 and 2023, what would a quadratic regression predict for 2024" as mentioned in the appendix. So quadratic regression makes it sound all fancy, but with two data points, it's literally just "extend the line straight". So the 2024 prediction is essentially meaningless.