3 ms·

I would feel better if this was derived from empirical data rather than just rhetoric. This seems super testable, no? There is probably a ton of data already i

I would feel better if this was derived from empirical data rather than just rhetoric. This seems super testable, no? There is probably a ton of data already in different industries with regards to productivity. Even if human talent have a Pareto distribution (which is not clear), the people employed by a company are a selected sub-set of that population, which would likely have a different distribution depending on how they are selected and the task at hand. I think that any of these simplified distributions are likely not generalizable across companies and industries (e.g. productivity of AWS or Google employees are likely not distributed like employees of MacDonalds or Wal*Mart because of the difference in hiring procedures and the nature of the tasks.) Get hard data within the companies and industry you are in and then you can make some arguments. Otherwise, I feel it is too easy to just be talking up a sand castle that has no solid footing.

groby_b 2y ago
> There is probably a ton of data already in different industries with regards to productivity. Uh. Not really. Our industry is notoriously bad at measuring productivity. And the bigger problem is that when we try to measure it - "performance review" - we like grading on a gaussian curve. We'll never know if that's correct because we put our thumb on the scale. An even bigger problem is that productivity is strongly influenced by completely non-technical factors. How enthusiastic are folks about what they are doing[1], how much variety do their tasks have [2], what are their peers like, etc. (Of course, that whole field of study has issues rooted in the inability to measure precisely as well) Ultimately, it's a squishy judgment applied by humans. [1] https://www.semanticscholar.org/paper/What-Predicts-Software-Developers%E2%80%99-Productivity-Murphy-Hill-Jaspan/494e91cca437cb791e5955c82d1899002e5272e2 https://www.semanticscholar.org/paper/What-Predicts-Software... [2] https://research.google/pubs/what-predicts-software-developers-productivity https://research.google/pubs/what-predicts-software-develope...
hinkley 2y ago
> I think that any of these simplified distributions are likely not generalizable across companies and industries It’s going to be multivariate statistics with dependent variables. The quality of non developers at company affects the quality of developers they can retain, and the quality of the developers you have affects the quality of developers you can recruit and improve. Almost all the people I’d want to work with again left my last employer before I did. You can take on more and more work yourself but it causes everyone around you to disengage. At some point you have to realize it’s more fruitful, emotionally and mathematically, to make coworkers produce one more unit of forward progress a month than to do it to yourself. Because it’s 2% for the team one way and 5-10% the other.
drcwpl 2y ago
Agree with you - although, rhetorically speaking, I have come across many instances which the author refers to "of low performers are 3x as common as high performers." This is unfortunate as I always think do your best, and as Tyler Cowen states - Average is Over. So agree it would have been way better to use empirical data to back up this claim especially.
KK7NIL 2y ago
The problem is that intellectual productivity is generally not possible to measure directly, so you instead end up with indirect measurements that assume a Gaussian distribution. IQ is famously Gaussian distributed... mainly because it's defined that way, not because human "intelligence" (good luck defining that) is Gaussian. If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc).
- bhouston 2y ago
  > so you instead end up with indirect measurements that assume a Gaussian distribution. 100%. I was going to write something similar. > If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc). Interesting, yeah, Elo is quite interesting. And one can view hiring in a company as something like selecting people for Elo above a certain score, but with some type of error distribution on top of that, probably Gaussian error. So what does a one sided Elo distribution look like with gaussian error in picking people above that Elo limit?
  - KK7NIL 2y ago
    Lichess has public population data (they use a modified version of Glicko-2 which is basically an updated version of Elo's system): https://lichess.org/stat/rating/distribution/blitz https://lichess.org/stat/rating/distribution/blitz It's basically a Gaussian with a very long right tail. Big caveat here is that these are the ratings of weekly active players. If we instead include casual players, I suspect we'd have something resembling a pareto distribution.
    - doctorpangloss 2y ago
      The big caveat is that it's trivial to measure the AIC, BIC and other quality of fit measurements for a distribution. If you think it's so and so distribution, go for it. In my experience in this specific case of chess rankings and in the broader case of test scores, skew-normal and log-normal have worse fits than plain Guassian. I have no idea why you would believe increasing the population would make this Gaussian distribution look Pareto, when the exact opposite is true - increasing populations make things look more Gaussian - in all natural circumstances.
Miraltar 2y ago
To me it says that our system is built on a reasonable but untested assumption (performance is a gaussian) and by replacing it with an equally reasonable assumption (performance is a pareto), suddenly our system looks stupid. It isn't really offering a solution but a new perspective
pama 2y ago
I thought that Bonus Content #1 and the references down the article were reasonably convincing. It would be great if large companies disclosed such details but it is unlikely.
wavemode 2y ago
> I would feel better if this was derived from empirical data rather than just rhetoric. This exact statement applies to the practice of Gaussian performance ranking. It is pure corporate politics, it isn't founded in sound statistics. The present author at least provides multiple sources of statistical evidence for their beliefs, if you read the footnotes.

groby_b 2y ago

> There is probably a ton of data already in different industries with regards to productivity. Uh. Not really. Our industry is notoriously bad at measuring productivity. And the bigger problem is that when we try to measure it - "performance review" - we like grading on a gaussian curve. We'll never know if that's correct because we put our thumb on the scale. An even bigger problem is that productivity is strongly influenced by completely non-technical factors. How enthusiastic are folks about what they are doing[1], how much variety do their tasks have [2], what are their peers like, etc. (Of course, that whole field of study has issues rooted in the inability to measure precisely as well) Ultimately, it's a squishy judgment applied by humans. [1] https://www.semanticscholar.org/paper/What-Predicts-Software-Developers%E2%80%99-Productivity-Murphy-Hill-Jaspan/494e91cca437cb791e5955c82d1899002e5272e2 https://www.semanticscholar.org/paper/What-Predicts-Software... [2] https://research.google/pubs/what-predicts-software-developers-productivity https://research.google/pubs/what-predicts-software-develope...

hinkley 2y ago

> I think that any of these simplified distributions are likely not generalizable across companies and industries It’s going to be multivariate statistics with dependent variables. The quality of non developers at company affects the quality of developers they can retain, and the quality of the developers you have affects the quality of developers you can recruit and improve. Almost all the people I’d want to work with again left my last employer before I did. You can take on more and more work yourself but it causes everyone around you to disengage. At some point you have to realize it’s more fruitful, emotionally and mathematically, to make coworkers produce one more unit of forward progress a month than to do it to yourself. Because it’s 2% for the team one way and 5-10% the other.

drcwpl 2y ago

Agree with you - although, rhetorically speaking, I have come across many instances which the author refers to "of low performers are 3x as common as high performers." This is unfortunate as I always think do your best, and as Tyler Cowen states - Average is Over. So agree it would have been way better to use empirical data to back up this claim especially.

KK7NIL 2y ago

The problem is that intellectual productivity is generally not possible to measure directly, so you instead end up with indirect measurements that assume a Gaussian distribution. IQ is famously Gaussian distributed... mainly because it's defined that way, not because human "intelligence" (good luck defining that) is Gaussian. If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc).

bhouston 2y ago
> so you instead end up with indirect measurements that assume a Gaussian distribution. 100%. I was going to write something similar. > If you look at board game Elo ratings (poor test for intelligence but we'll ignore that), they do not follow a Gaussian distribution, even though Elo assumes a Gaussian distribution for game outcomes (but not the population). So that's good evidence that aptitude/skill in intellectual subjects isn't Gaussian (but it's also not Pareto iirc). Interesting, yeah, Elo is quite interesting. And one can view hiring in a company as something like selecting people for Elo above a certain score, but with some type of error distribution on top of that, probably Gaussian error. So what does a one sided Elo distribution look like with gaussian error in picking people above that Elo limit?
- KK7NIL 2y ago
  Lichess has public population data (they use a modified version of Glicko-2 which is basically an updated version of Elo's system): https://lichess.org/stat/rating/distribution/blitz https://lichess.org/stat/rating/distribution/blitz It's basically a Gaussian with a very long right tail. Big caveat here is that these are the ratings of weekly active players. If we instead include casual players, I suspect we'd have something resembling a pareto distribution.
  - doctorpangloss 2y ago
    The big caveat is that it's trivial to measure the AIC, BIC and other quality of fit measurements for a distribution. If you think it's so and so distribution, go for it. In my experience in this specific case of chess rankings and in the broader case of test scores, skew-normal and log-normal have worse fits than plain Guassian. I have no idea why you would believe increasing the population would make this Gaussian distribution look Pareto, when the exact opposite is true - increasing populations make things look more Gaussian - in all natural circumstances.

Miraltar 2y ago

To me it says that our system is built on a reasonable but untested assumption (performance is a gaussian) and by replacing it with an equally reasonable assumption (performance is a pareto), suddenly our system looks stupid. It isn't really offering a solution but a new perspective

pama 2y ago

I thought that Bonus Content #1 and the references down the article were reasonably convincing. It would be great if large companies disclosed such details but it is unlikely.

wavemode 2y ago

> I would feel better if this was derived from empirical data rather than just rhetoric. This exact statement applies to the practice of Gaussian performance ranking. It is pure corporate politics, it isn't founded in sound statistics. The present author at least provides multiple sources of statistical evidence for their beliefs, if you read the footnotes.