3 ms·

Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure

Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure with the intent to improve how your team works. If a manager can measure at the team level and open up visibility into the development process, they can hopefully find where things get frustrating (ex. waiting for someone to review a PR). That said, there seem to be more mature tools out there than OKAY's beta. There's a discord server called dev interrupted that talks about this stuff a lot.

sdesol 6y ago
I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rvfx https://imgur.com/DK7rvfx (gitlab) You'll find they both have a large cluster of developers in zone 2. For developers to exist in zone 2, they have to have medium to high impact on the code that they worked on, but not clash with others. If you look at the vuejs-next repository https://imgur.com/eDAOyPW https://imgur.com/eDAOyPW (vuejs) You can see it's actually a pretty fragile project, since Evan is responsible for pretty much everything. Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. Edit: The metrics that I'm showing is limited to the last 90 days for Typescript, Javascript and CSS code.
- majormajor 6y ago
  > Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. This is a dangerous conclusion, especially for a business. If you're in pure maintenance mode, maybe... but otherwise... You want people who can pitch in anywhere, who can fix things rapidly, who can build new solutions quickly when required, and who know your business inside and out. You just don't want knowledge siloed there, so you want to make sure other people are also on the path to being expert on the various areas.
  - sdesol 6y ago
    Context obviously matters and I think it is important to understand what I mean by "Very High" and "High" impact employees. High impact employees can still do everything by my definition, it's just that they aren't doing everything. Obviously some business/projects do not have the luxury of attracting lots of talented employees, so "Very High" impact employees are inevitable. If you look at the busfactor stats in the deep dive section https://imgur.com/tiGFmFf https://imgur.com/tiGFmFf The number of files that are being changed with only one author is 419 (or about 25% of all the files changed in the 90 days window). So 75% of the files changed in the 90 days window have two or more contributors, so I think those working on the vscode project aren't being siloed (based on my quick observations).
- visarga 6y ago
  > you don't want a single individual to stand out A study from 2016 at Google discovered that team effectiveness is related to the opportunity for "equal speaking". Similar to your conclusion. https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html https://www.nytimes.com/2016/02/28/magazine/what-google-lear...
- didibus 6y ago
  What's missing from your assessment is how good the project itself is. Maybe Evan is a liability, but he might also be the reason why VueJS is popular and valued in the first place. It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn. If you took VueJS, and had a team at Microsoft take it over from Evan, it too would live on. So I don't know that your explanation for the "risk" here has anything to do with "very high impact" and more to do with a project being maintained by a community of backers, on people's free time, and a project maintained by a company that hires developers to work on it full time.
  - sdesol 6y ago
    > So I don't know that your explanation for the "risk" here has anything to do with "very high impact" When talking about risk, I mean immediate as opposed to long term. Obviously, if you pay people to take over vuejs, they can, provided they are qualified that is. But there is still the ramp up period and the risk of losing undocumented knowledge that Evan has, specifically with his understanding of "what doesn't work". > It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn. The vscode and gitlab project are included because they provide good data points for very fast moving projects, that can be used to help understand closed source software development. Also the development pattern behind vuejs certainly exists in the closed source world as well.
    - didibus 6y ago
      I think my criticism is I'm not really seeing where in your data is risk or quality or success of the project quantified and correlated to this "very high impact" conclusion. It seems you're inferring this "risk" from a qualitative perspective. Like hypothetically, we can imagine VueJS being more at risk of being abandoned because there's only one big maintainer. But VueJS hasn't been abandoned, and is doing well. So does the data support this hypothetical? And I'm also not sure how abandonment relates to productivity. If GitLab the company folds, that project will probably stop being developed and be abandoned, same as if Evan stops working on VueJS. Which one is more likely? No one knows, but companies can abandon an open source product probably just as much as community contributors. So I feel it's more about which one is more likely to be picked up after the current maintainers abandon it.
- deleted 6y ago
  [deleted]
- michaelt 6y ago
  > Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. If you're trying to sell this concept to developers, you might want to change "discourage" to "hire more than one"
  - sdesol 6y ago
    Ideally you would like to hire more than one, but I think my use of the word "discourage", was not the right one. What I ultimately wanted to say was, you want a balanced team, where no individual development pattern sticks out. A very high impact employee could easily be the result of having multiple poor employees, improper planning, and so forth.

sdesol 6y ago

I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rvfx https://imgur.com/DK7rvfx (gitlab) You'll find they both have a large cluster of developers in zone 2. For developers to exist in zone 2, they have to have medium to high impact on the code that they worked on, but not clash with others. If you look at the vuejs-next repository https://imgur.com/eDAOyPW https://imgur.com/eDAOyPW (vuejs) You can see it's actually a pretty fragile project, since Evan is responsible for pretty much everything. Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. Edit: The metrics that I'm showing is limited to the last 90 days for Typescript, Javascript and CSS code.

majormajor 6y ago
> Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. This is a dangerous conclusion, especially for a business. If you're in pure maintenance mode, maybe... but otherwise... You want people who can pitch in anywhere, who can fix things rapidly, who can build new solutions quickly when required, and who know your business inside and out. You just don't want knowledge siloed there, so you want to make sure other people are also on the path to being expert on the various areas.
- sdesol 6y ago
  Context obviously matters and I think it is important to understand what I mean by "Very High" and "High" impact employees. High impact employees can still do everything by my definition, it's just that they aren't doing everything. Obviously some business/projects do not have the luxury of attracting lots of talented employees, so "Very High" impact employees are inevitable. If you look at the busfactor stats in the deep dive section https://imgur.com/tiGFmFf https://imgur.com/tiGFmFf The number of files that are being changed with only one author is 419 (or about 25% of all the files changed in the 90 days window). So 75% of the files changed in the 90 days window have two or more contributors, so I think those working on the vscode project aren't being siloed (based on my quick observations).
visarga 6y ago
> you don't want a single individual to stand out A study from 2016 at Google discovered that team effectiveness is related to the opportunity for "equal speaking". Similar to your conclusion. https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html https://www.nytimes.com/2016/02/28/magazine/what-google-lear...
didibus 6y ago
What's missing from your assessment is how good the project itself is. Maybe Evan is a liability, but he might also be the reason why VueJS is popular and valued in the first place. It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn. If you took VueJS, and had a team at Microsoft take it over from Evan, it too would live on. So I don't know that your explanation for the "risk" here has anything to do with "very high impact" and more to do with a project being maintained by a community of backers, on people's free time, and a project maintained by a company that hires developers to work on it full time.
- sdesol 6y ago
  > So I don't know that your explanation for the "risk" here has anything to do with "very high impact" When talking about risk, I mean immediate as opposed to long term. Obviously, if you pay people to take over vuejs, they can, provided they are qualified that is. But there is still the ramp up period and the risk of losing undocumented knowledge that Evan has, specifically with his understanding of "what doesn't work". > It also seems pretty strange to me to count VSCode and GitLab in there, because those are worked on by companies with teams behind them that get paid and which will have constant churn. The vscode and gitlab project are included because they provide good data points for very fast moving projects, that can be used to help understand closed source software development. Also the development pattern behind vuejs certainly exists in the closed source world as well.
  - didibus 6y ago
    I think my criticism is I'm not really seeing where in your data is risk or quality or success of the project quantified and correlated to this "very high impact" conclusion. It seems you're inferring this "risk" from a qualitative perspective. Like hypothetically, we can imagine VueJS being more at risk of being abandoned because there's only one big maintainer. But VueJS hasn't been abandoned, and is doing well. So does the data support this hypothetical? And I'm also not sure how abandonment relates to productivity. If GitLab the company folds, that project will probably stop being developed and be abandoned, same as if Evan stops working on VueJS. Which one is more likely? No one knows, but companies can abandon an open source product probably just as much as community contributors. So I feel it's more about which one is more likely to be picked up after the current maintainers abandon it.
deleted 6y ago
[deleted]
michaelt 6y ago
> Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. If you're trying to sell this concept to developers, you might want to change "discourage" to "hire more than one"
- sdesol 6y ago
  Ideally you would like to hire more than one, but I think my use of the word "discourage", was not the right one. What I ultimately wanted to say was, you want a balanced team, where no individual development pattern sticks out. A very high impact employee could easily be the result of having multiple poor employees, improper planning, and so forth.