6 ms·

Has anyone published some kind of quantified, "hard" comparison/survey about the relative productivity of programmers? It would be very interesting with a clea

by unwind 5y ago

Has anyone published some kind of quantified, "hard" comparison/survey about the relative productivity of programmers? It would be very interesting with a clean "we asked n programmers to solve this"-type problem, and then being able to compare the solutions. Uh, did I just sound like I wanted to work in recruiting? #fail

flakiness 5y ago
You might like this site https://neverworkintheory.org/date/ https://neverworkintheory.org/date/ It reviews academic papers in that kind (empirical software engineering).
crispyambulance 5y ago
No one has figured this out-- not for programmers and not for any other profession. It's certainly possible for individuals to develop good or bad reputations, but to be able to accurately evaluate how anyone will do in a job in the absence of other data is very much hit-or-miss. The most elite orgs might LOOK LIKE they figured it out, but how much of that is really their process vs just being able to attract talent because of compensation or reputation? How much of it is more a matter of being able to churn through people and replace people that get hired but can't hack it once they're in? How much of it is skillful use of professional referrals? > It would be very interesting with a clean "we asked n programmers to solve this"-type problem [...] It's been done, right? There's all those whiteboard interviews that highly resourced orgs study endlessly. There's a whole industry around preparing for these things. What do we have to show for it as an industry? Not much. If the stories are to be believed, folks are still flunking "fizz-buzz" at alarming rates.
theamk 5y ago
There is a link to this right in the post. In fact, it is the first link in the post. Here it is: https://href.li/?https://third-bit.com/talks/greatest-hits/?utm_campaign=CLICKWORTHY&utm_medium=email&utm_source=Revue%20newsletter#28 https://href.li/?https://third-bit.com/talks/greatest-hits/?...
Izkata 5y ago
About a decade ago my company had a thing set up where were could complete coding challenges and other devs would rate them anonymously (so you couldn't rate a specific person highly unless you recognized their style). Once you completed a challenge you could see others' submissions and a ranking of how much someone's solutions were liked. Ordering was fairly consistent, with a few people recognizably always being in the top third or so (out of around 15-25 people who bothered to try the challenges, IIRC). The original purpose of these challenges were to help people learn python, as we'd just officially moved to it not long before I joined, so to give an idea of the complexity involved, the only one I can remember was "implement a brainfuck interpreter". It was really interesting to see the different ways people organized code that did the same thing.
- unwind 5y ago
  Wow, that sounds like a lot of fun and also (if the culture is right) like a great learning opportunity for many on the team. Since this was done at work, it kind of suggests a good culture, obviously. Thanks for sharing!
ratww 5y ago
At several companies I worked for, by using Scrum Points, it was always visible that the most experienced devs were able to regularly deliver a multiple (definitely around 10x) more points than interns and about 2x more than the average dev. Not a perfect metric by any means, of course, but definitely quantifiable and observable. Metrics for LOC and commits/PRs made also followed a similar pattern. Again, not perfect metrics, but it's what we had. Like another poster said [0], it's not really 10x the average, it's 10x when comparing best vs worst. [0] https://news.ycombinator.com/item?id=31074087 https://news.ycombinator.com/item?id=31074087
- deleted 5y ago
  [deleted]
- torginus 5y ago
  The problem with this metric is it can be gamed. We had something similar, and there was a huge 'meta' over who gets the shiny new features, and who has to fix the legacy code which was like Jenga, and a single misstep could break thousands of tests, and writing code was like playing minesweeper. The people who won the 'meta', got to be the most productive. Thing is, probably the guy who fixed a bug in the legacy hairball by writing 10 lines of code produced tons of value for the organization (those unit tests weren't for show, they were customer use cases), while the other one only produced some, if the fancy new feature got adopted.
- Trasmatta 5y ago
  Are you certain the devs getting 2x more points than the average dev weren't just consistently picking the easier stories? Or estimating story points vastly differently than other members of the team? In my experience, story points can be worse than no estimation. They give the illusion of insight, but in reality there's so much noise and inconsistency in point estimation, that they're almost never accurate measures.
  - ratww 5y ago
    > Are you certain the devs getting 2x more points than the average dev weren't just consistently picking the easier stories? Your objections make no sense. If a story is easy but the number of points is not reflecting the ease, then the estimation is wrong, period. If there is someone on a team with such an estimation superpower so that they're able to know which tickets are easy, then just make this person estimate all the tickets. Problem solved. You either accept that someone is more productive, or you accept the estimation you performed is wrong by a factor of 2x. You can't have it both ways. > Or estimating story points vastly differently than other members of the team? In all the teams I worked the estimation was done by the entire team. If it's not being done by the whole team, then there's no real baseline to compare the work being done by two people, period. > In my experience, story points can be worse than no estimation. They give the illusion of insight, but in reality there's so much noise and inconsistency in point estimation, that they're almost never accurate measures. The point of story points is not to give extremely accurate measurements, but rather to inform you of the amount of work a team is able to do in a period of time (often a Sprint).

by unwind 5y ago

flakiness 5y ago

You might like this site https://neverworkintheory.org/date/ https://neverworkintheory.org/date/ It reviews academic papers in that kind (empirical software engineering).

crispyambulance 5y ago

No one has figured this out-- not for programmers and not for any other profession. It's certainly possible for individuals to develop good or bad reputations, but to be able to accurately evaluate how anyone will do in a job in the absence of other data is very much hit-or-miss. The most elite orgs might LOOK LIKE they figured it out, but how much of that is really their process vs just being able to attract talent because of compensation or reputation? How much of it is more a matter of being able to churn through people and replace people that get hired but can't hack it once they're in? How much of it is skillful use of professional referrals? > It would be very interesting with a clean "we asked n programmers to solve this"-type problem [...] It's been done, right? There's all those whiteboard interviews that highly resourced orgs study endlessly. There's a whole industry around preparing for these things. What do we have to show for it as an industry? Not much. If the stories are to be believed, folks are still flunking "fizz-buzz" at alarming rates.

theamk 5y ago

There is a link to this right in the post. In fact, it is the first link in the post. Here it is: https://href.li/?https://third-bit.com/talks/greatest-hits/?utm_campaign=CLICKWORTHY&utm_medium=email&utm_source=Revue%20newsletter#28 https://href.li/?https://third-bit.com/talks/greatest-hits/?...

Izkata 5y ago

About a decade ago my company had a thing set up where were could complete coding challenges and other devs would rate them anonymously (so you couldn't rate a specific person highly unless you recognized their style). Once you completed a challenge you could see others' submissions and a ranking of how much someone's solutions were liked. Ordering was fairly consistent, with a few people recognizably always being in the top third or so (out of around 15-25 people who bothered to try the challenges, IIRC). The original purpose of these challenges were to help people learn python, as we'd just officially moved to it not long before I joined, so to give an idea of the complexity involved, the only one I can remember was "implement a brainfuck interpreter". It was really interesting to see the different ways people organized code that did the same thing.

unwind 5y ago
Wow, that sounds like a lot of fun and also (if the culture is right) like a great learning opportunity for many on the team. Since this was done at work, it kind of suggests a good culture, obviously. Thanks for sharing!

ratww 5y ago

At several companies I worked for, by using Scrum Points, it was always visible that the most experienced devs were able to regularly deliver a multiple (definitely around 10x) more points than interns and about 2x more than the average dev. Not a perfect metric by any means, of course, but definitely quantifiable and observable. Metrics for LOC and commits/PRs made also followed a similar pattern. Again, not perfect metrics, but it's what we had. Like another poster said [0], it's not really 10x the average, it's 10x when comparing best vs worst. [0] https://news.ycombinator.com/item?id=31074087 https://news.ycombinator.com/item?id=31074087

deleted 5y ago
[deleted]
torginus 5y ago
The problem with this metric is it can be gamed. We had something similar, and there was a huge 'meta' over who gets the shiny new features, and who has to fix the legacy code which was like Jenga, and a single misstep could break thousands of tests, and writing code was like playing minesweeper. The people who won the 'meta', got to be the most productive. Thing is, probably the guy who fixed a bug in the legacy hairball by writing 10 lines of code produced tons of value for the organization (those unit tests weren't for show, they were customer use cases), while the other one only produced some, if the fancy new feature got adopted.
Trasmatta 5y ago
Are you certain the devs getting 2x more points than the average dev weren't just consistently picking the easier stories? Or estimating story points vastly differently than other members of the team? In my experience, story points can be worse than no estimation. They give the illusion of insight, but in reality there's so much noise and inconsistency in point estimation, that they're almost never accurate measures.
- ratww 5y ago
  > Are you certain the devs getting 2x more points than the average dev weren't just consistently picking the easier stories? Your objections make no sense. If a story is easy but the number of points is not reflecting the ease, then the estimation is wrong, period. If there is someone on a team with such an estimation superpower so that they're able to know which tickets are easy, then just make this person estimate all the tickets. Problem solved. You either accept that someone is more productive, or you accept the estimation you performed is wrong by a factor of 2x. You can't have it both ways. > Or estimating story points vastly differently than other members of the team? In all the teams I worked the estimation was done by the entire team. If it's not being done by the whole team, then there's no real baseline to compare the work being done by two people, period. > In my experience, story points can be worse than no estimation. They give the illusion of insight, but in reality there's so much noise and inconsistency in point estimation, that they're almost never accurate measures. The point of story points is not to give extremely accurate measurements, but rather to inform you of the amount of work a team is able to do in a period of time (often a Sprint).