3 ms·

N = 16 developers. Is this enough to draw any meaningful conclusions?

• sarchertech 1y ago
  That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection. If you are looking for a 0.1% increase in productivity, then 16 is too small.
  • biophysboy 1y ago
    Well it depends on the variance of the random variable itself. You're right that with big, obvious effects, a larger n is less "necessary". I could see individuals having very different "productivities", especially when the idea is flattened down to completion time.
    • sarchertech 1y ago
      Individuals do have very different productivities, but they are measuring the productivity difference across a single individual.
  • AvAn12 1y ago
    “A quarter of the participants saw increased performance, 3/4 saw reduced performance.” So I think any conclusions drawn on these 16 people doesn’t signify much one way or the other. Cool paper but how is this anything other than a null finding?
    • tripletao 1y ago
      They show a 95% CI excluding zero in Figure 1. By the usual standards of social science, that's not a null finding. They give their methodology in Appendix D. For intuition on why it's insufficient to consider N alone, I assume e.g. that you'd greatly increase your belief that a coin was unfair long before 16 consecutive heads--as already noted, the size of the effect also matters. That relationship isn't intuitive in general, and attempts to replace the math with feelings tend to fail.
• phyzome 1y ago
  Or N = 246 tasks.