4 ms·

Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.

by Jabrov 1y ago

IshKebab 1y ago
They paid the developers about $75k in total to do this so I wouldn't hold your breath!
- barbazoo 1y ago
  That's a lot of money for many of us. Do you know those folks were in a HCOL area?
  - IshKebab 1y ago
    No idea. They don't say who they were; just random popular GitHub projects. To be clear it wasn't $75k each.
    - narush 1y ago
      You can see a list of repositories with participating developers in the appendix! Section G.7. Paper is here: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
  - mapt 1y ago
    It isn't a lot of money for industry research. Changes of +-40% in productivity are an enormous advantage/disadvantage for a large tech company moving tens of billions of dollars a year in cashflow through a pipeline that their software engineers built.
- lawlessone 1y ago
  Neat, how to sign up??
  - IshKebab 1y ago
    Go back in time, create a popular github repo with lots of stars, be lucky.
  - asdff 1y ago
    I see these things posted on linkedin. Usually asking $40/hr though. But essentially the same thing as the OP outlines: you do some domain related task assigned either with or without an AI tool. Check linked in. They will have really vague titles like "data scientist" though even though that's not what is being described, its study subject. Maybe set 40/hr as a filter on linkedin and see if you can get a few to come up.
narush 1y ago
Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.
- deleted 1y ago
  [deleted]
- biophysboy 1y ago
  Did you compare the variance within individuals (due to treatment) to the variance between individuals (due to other stuff)?

Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.

by Jabrov 1y ago

Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.

IshKebab 1y ago

They paid the developers about $75k in total to do this so I wouldn't hold your breath!

barbazoo 1y ago
That's a lot of money for many of us. Do you know those folks were in a HCOL area?
- IshKebab 1y ago
  No idea. They don't say who they were; just random popular GitHub projects. To be clear it wasn't $75k each.
  - narush 1y ago
    You can see a list of repositories with participating developers in the appendix! Section G.7. Paper is here: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
- mapt 1y ago
  It isn't a lot of money for industry research. Changes of +-40% in productivity are an enormous advantage/disadvantage for a large tech company moving tens of billions of dollars a year in cashflow through a pipeline that their software engineers built.
lawlessone 1y ago
Neat, how to sign up??
- IshKebab 1y ago
  Go back in time, create a popular github repo with lots of stars, be lucky.
- asdff 1y ago
  I see these things posted on linkedin. Usually asking $40/hr though. But essentially the same thing as the OP outlines: you do some domain related task assigned either with or without an AI tool. Check linked in. They will have really vague titles like "data scientist" though even though that's not what is being described, its study subject. Maybe set 40/hr as a filter on linkedin and see if you can get a few to come up.

narush 1y ago

Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.

deleted 1y ago
[deleted]
biophysboy 1y ago
Did you compare the variance within individuals (due to treatment) to the variance between individuals (due to other stuff)?