13 ms·

I have worked in team with some kernel developers at Samsung, years ago, so let me put some perspective for those who do not understand the dynamics. In some c

I have worked in team with some kernel developers at Samsung, years ago, so let me put some perspective for those who do not understand the dynamics. In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success. As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or nonsensical. Because in the very end almost nobody sees the amount of value that a company creates for Linux users. What everybody is looking at is that "Company X is nth on the list of biggest kernel contributors". So "KPI grabbing" is meant to mean the practice of not caring for the value or quality of your contributions but rather for the count of the commits or LoC you can get accepted. This is damaging to Linux kernel developer community because it just creates work for maintainers without creating much or any value at all. These maintainers are absolutely critical resource for the project and their number and availability translates directly to how much stuff can actually be done. So no wonder people are irked by it.

newsclues 5y ago
Maintainers need their own metrics that are effective, and publish them and work to have partners use them.
- duxup 5y ago
  I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law. This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.
  - newsclues 5y ago
    I’m not knowledgeable enough to define what good work looks like but, simply, whenever maintainers see good work, give that user a golden star. People will work for the carrot, so give carrots for the work maintainers want to see.
    - PixelOfDeath 5y ago
      Then you get them into trying to get their own people into maintainer positions to let the golden stars flow. Maybe also be more hesitant to give out this starts to the "wrong" people from other companies. Not like there already is enough politic bullshit in open source.
- lmilcin 5y ago
  The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue. But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.
  - newsclues 5y ago
    Yes, so either accept the status quo or create positive change. Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.
    - trasz 5y ago
      What for? Metrics for metrics sake are a waste of time. The solution is to avoid wasting time on metrics, which is precisely what happened here.
    - ori_b 5y ago
      Sometimes, throwing trash into the trash bin is positive change. You don't need to put new trash in as a replacement.
  - qsmlfkqjsd 5y ago
    > The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.
    - lmilcin 5y ago
      It is not about any single patch, you can't assess value of any patch objectively. For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions. In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?
- tgsovlerkhgsel 5y ago
  Doesn't work. "When a measure becomes a target, it ceases to be a good measure." You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.
  - acover 5y ago
    Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it. Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.
    - simonh 5y ago
      Because that would be a huge time sink for the maintainers as well. They’re the only ones qualified to evaluate quality and they have better things to do with the time it would take. This problem is created by these companies and it’s their problem to solve.
    - jart 5y ago
      I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.
      - sdesol 5y ago
        > I tend to look on commit count as a negative when reviewing open source projects. I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment. Take the following for example: https://public-001.gitsense.com/insights/github/repos?p=impact&r=github%3Amicrosoft%2Fvscode&v=repo https://public-001.gitsense.com/insights/github/repos?p=impa... by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace. Software metrics in general isn't the issue, it's the lack of context around them that is the issue. Disclaimer: I'm the creator of the tool that I'm linking to
radiator 5y ago
Maybe a panel of maintainers could publish a quarterly review with attribution/thanks to the most important contributors. No metrics to game, only pure human expert opinions.
- exikyut 5y ago
  [Generic hand-waving about how LWN kinda does this, in a high-signal manner unfriendly to statistical summation]
- everybodyknows 5y ago
  Maintainers aren’t interested in taking on extra chores. However, important contributors already do receive attribution, in the format of news articles, here: https://lwn.net/Archives/ https://lwn.net/Archives/
- jldugger 5y ago
  IMO part of the problem is that Greg KH publishes this annual 'state of the kernel' that is just a list of metrics to game. Which feels like it was started to shame non-RH companies to contribute more often. So, kinda maybe beware what you wish for? Example: https://www.youtube.com/watch?v=SIQr2-Dh0es https://www.youtube.com/watch?v=SIQr2-Dh0es
jart 5y ago
Could someone ELI5 to me what KPI even stands for? Does that mean like they're like the shock troopers of open source?
- neilalexander 5y ago
  Key Performance Indicator.
- martincmartin 5y ago
  https://en.wikipedia.org/wiki/Performance_indicator https://en.wikipedia.org/wiki/Performance_indicator Engineer's performance is measured, in part, by number of kernel commits. So, the number of kernel commits is a key performance indicator for the developer.
- NikolaNovak 5y ago
  KPI = Key Performance Indicator = "thing we decided is important for us to measure and increase" As such exactly what it is, will depend on company/industry/market/team/project/etc. It may be revenue or units sold or widgets made or turnover or customer satisfaction or days without accident etc. In principle, it communicates to teams what is important to business and helps everybody focus and sync.... with usual real-world caveats and implementation risks. I imagine here, for a large-company linux team, a "KPI" was "how many commits you made" - they measure it, they track it, and reward/recognize/punish team members based on it. So people start working toward increasing the KPI in any feasible way - if you only want me to increase the KPI of commits done, OK, fine, I'll do some commits.
  - falcor84 5y ago
    >thing we decided is important for us to measure and increase This is a great explanation, and I just wanted to nitpick that for some KPI's (e.g. percentage of defective components, or latency), you would aim to "decrease" the KPI, and in some rare instances, you'd want to keep the KPI within a range (e.g. resource utilization, or inventory size, when you want a predefined amount of buffer).
- alxlaz 5y ago
faeyanpiraat 5y ago
It's like the days when outsourcing software development to india was new, and they charged for number of lines of code written.
- protomyth 5y ago
  IBM apparently went that way too https://www.youtube.com/watch?v=kHI7RTKhlz0 https://www.youtube.com/watch?v=kHI7RTKhlz0
wyager 5y ago
This is why ousting Linus puts the long-term health of the kernel at risk. It’s very difficult and expensive, at a personal and professional level, to tell colleagues to “fuck off” if they are submitting low quality garbage for reasons related to their salary. Linux was Linus’s baby, he had nearly absolute control, and he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons. Now, if the people approving patches are just normal employees who can get fired if a kernel sponsor complains enough, it’s going to be much harder for them to successfully deflect all the bullshit without ever caving in.
- otikik 5y ago
  I think the email was completely effective in denouncing a behavior. Telling people to "fuck off" isn't required (nor desired, in my opinion). Linux health happened *despite* Linus' manners, not because of them.
  - lmilcin 5y ago
    It is naive to claim you can tell which part of Linus is and which isn't contributing to the success. For one, I think it might be possible Linus's no-nonsense attitude is for the better of the organization as people who can't work with it are leaving causing Linux developers, on average, to be more no-nonsense and also able to coexist better with other no-nonsense people. But, it is my conjecture only. We would never know how Linux would fare if Linus was another guy.
    - fao_ 5y ago
      > It is naive to claim you can tell which part of Linus is and which isn't contributing to the success. Wasn't the other person doing exactly the same, just on the "pro-asshole" side of the argument?
      - yebyen 5y ago
        It is probably easier and more defensible to look at a thing that is, and say "why is it this way" than to see something that isn't and ask, "why isn't it the way I want it to be?" These are logically not the same structure of argument. I'm not making any character judgements about people involved in this discussion, or on the LKML either for that matter.
moonshinefe 5y ago
Thanks for your perspective I didn't actually realize that dynamic was at play (at an organizational level, anyway, I'm well aware individuals game # of contributions). These organizations pushing trivial/garbage patches and wasting maintainers' time absolutely have to be called out by the community as unacceptable, so I'm glad to see the post. This is a massive waste of time, time which open source contributors/maintainers often are already short on.
cjohansson 5y ago
Sounds like the typical issue of business intelligence using quantity over quality to rate work performance
croes 5y ago
"When a measure becomes a target, it ceases to be a good measure."
- thih9 5y ago
  Corollary: "Businesses that base their growth on A/B testing only, turn into gambling or pornography".
  - Seirdy 5y ago
    Relevant xkcd: "Video Content" https://xkcd.com/1804/ https://xkcd.com/1804/ A/B and telemetry-worship reminds me of something I posted on Fedi earlier. Pasting it below: --- "Guys, I have an idea. Let's make a browser optimized for the kind of people who don't opt out of telemetry, and market it to privacy-conscious people." -- Top minds at Mozilla Corp If you market to privacy-conscious people and you base all your decisions on data from telemetry, you're gonna get a very skewed perspective. You'll see disproportionate representation from fans (people who want to enable telemetry because they love Mozilla and want to help, and will rationalize any decision Mozilla makes) and very non-technical users who don't know what telemetry is and how opt-in/out works. Technical users who disable stuff like telemetry, analytics, etc. are going to look like a rounding error. This explains a lot of Firefox's changes.
    - AwaAwa 5y ago
      This is enlightening and depressing. It's sad to see the Fox let itself go.
- squabble 5y ago
  I sometimes wonder if something like this has happened with money.
  - deleted 5y ago
    [deleted]

newsclues 5y ago

Maintainers need their own metrics that are effective, and publish them and work to have partners use them.

duxup 5y ago
I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law. This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.
- newsclues 5y ago
  I’m not knowledgeable enough to define what good work looks like but, simply, whenever maintainers see good work, give that user a golden star. People will work for the carrot, so give carrots for the work maintainers want to see.
  - PixelOfDeath 5y ago
    Then you get them into trying to get their own people into maintainer positions to let the golden stars flow. Maybe also be more hesitant to give out this starts to the "wrong" people from other companies. Not like there already is enough politic bullshit in open source.
lmilcin 5y ago
The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue. But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.
- newsclues 5y ago
  Yes, so either accept the status quo or create positive change. Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.
  - trasz 5y ago
    What for? Metrics for metrics sake are a waste of time. The solution is to avoid wasting time on metrics, which is precisely what happened here.
  - ori_b 5y ago
    Sometimes, throwing trash into the trash bin is positive change. You don't need to put new trash in as a replacement.
- qsmlfkqjsd 5y ago
  > The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.
  - lmilcin 5y ago
    It is not about any single patch, you can't assess value of any patch objectively. For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions. In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?
tgsovlerkhgsel 5y ago
Doesn't work. "When a measure becomes a target, it ceases to be a good measure." You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.
- acover 5y ago
  Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it. Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.
  - simonh 5y ago
    Because that would be a huge time sink for the maintainers as well. They’re the only ones qualified to evaluate quality and they have better things to do with the time it would take. This problem is created by these companies and it’s their problem to solve.
  - jart 5y ago
    I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.
    - sdesol 5y ago
      > I tend to look on commit count as a negative when reviewing open source projects. I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment. Take the following for example: https://public-001.gitsense.com/insights/github/repos?p=impact&r=github%3Amicrosoft%2Fvscode&v=repo https://public-001.gitsense.com/insights/github/repos?p=impa... by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace. Software metrics in general isn't the issue, it's the lack of context around them that is the issue. Disclaimer: I'm the creator of the tool that I'm linking to

radiator 5y ago

Maybe a panel of maintainers could publish a quarterly review with attribution/thanks to the most important contributors. No metrics to game, only pure human expert opinions.

exikyut 5y ago
[Generic hand-waving about how LWN kinda does this, in a high-signal manner unfriendly to statistical summation]
everybodyknows 5y ago
Maintainers aren’t interested in taking on extra chores. However, important contributors already do receive attribution, in the format of news articles, here: https://lwn.net/Archives/ https://lwn.net/Archives/
jldugger 5y ago
IMO part of the problem is that Greg KH publishes this annual 'state of the kernel' that is just a list of metrics to game. Which feels like it was started to shame non-RH companies to contribute more often. So, kinda maybe beware what you wish for? Example: https://www.youtube.com/watch?v=SIQr2-Dh0es https://www.youtube.com/watch?v=SIQr2-Dh0es

jart 5y ago

Could someone ELI5 to me what KPI even stands for? Does that mean like they're like the shock troopers of open source?

neilalexander 5y ago
Key Performance Indicator.
martincmartin 5y ago
https://en.wikipedia.org/wiki/Performance_indicator https://en.wikipedia.org/wiki/Performance_indicator Engineer's performance is measured, in part, by number of kernel commits. So, the number of kernel commits is a key performance indicator for the developer.
NikolaNovak 5y ago
KPI = Key Performance Indicator = "thing we decided is important for us to measure and increase" As such exactly what it is, will depend on company/industry/market/team/project/etc. It may be revenue or units sold or widgets made or turnover or customer satisfaction or days without accident etc. In principle, it communicates to teams what is important to business and helps everybody focus and sync.... with usual real-world caveats and implementation risks. I imagine here, for a large-company linux team, a "KPI" was "how many commits you made" - they measure it, they track it, and reward/recognize/punish team members based on it. So people start working toward increasing the KPI in any feasible way - if you only want me to increase the KPI of commits done, OK, fine, I'll do some commits.
- falcor84 5y ago
  >thing we decided is important for us to measure and increase This is a great explanation, and I just wanted to nitpick that for some KPI's (e.g. percentage of defective components, or latency), you would aim to "decrease" the KPI, and in some rare instances, you'd want to keep the KPI within a range (e.g. resource utilization, or inventory size, when you want a predefined amount of buffer).
alxlaz 5y ago

faeyanpiraat 5y ago

It's like the days when outsourcing software development to india was new, and they charged for number of lines of code written.

protomyth 5y ago
IBM apparently went that way too https://www.youtube.com/watch?v=kHI7RTKhlz0 https://www.youtube.com/watch?v=kHI7RTKhlz0

wyager 5y ago

This is why ousting Linus puts the long-term health of the kernel at risk. It’s very difficult and expensive, at a personal and professional level, to tell colleagues to “fuck off” if they are submitting low quality garbage for reasons related to their salary. Linux was Linus’s baby, he had nearly absolute control, and he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons. Now, if the people approving patches are just normal employees who can get fired if a kernel sponsor complains enough, it’s going to be much harder for them to successfully deflect all the bullshit without ever caving in.

otikik 5y ago
I think the email was completely effective in denouncing a behavior. Telling people to "fuck off" isn't required (nor desired, in my opinion). Linux health happened *despite* Linus' manners, not because of them.
- lmilcin 5y ago
  It is naive to claim you can tell which part of Linus is and which isn't contributing to the success. For one, I think it might be possible Linus's no-nonsense attitude is for the better of the organization as people who can't work with it are leaving causing Linux developers, on average, to be more no-nonsense and also able to coexist better with other no-nonsense people. But, it is my conjecture only. We would never know how Linux would fare if Linus was another guy.
  - fao_ 5y ago
    > It is naive to claim you can tell which part of Linus is and which isn't contributing to the success. Wasn't the other person doing exactly the same, just on the "pro-asshole" side of the argument?
    - yebyen 5y ago
      It is probably easier and more defensible to look at a thing that is, and say "why is it this way" than to see something that isn't and ask, "why isn't it the way I want it to be?" These are logically not the same structure of argument. I'm not making any character judgements about people involved in this discussion, or on the LKML either for that matter.

moonshinefe 5y ago

Thanks for your perspective I didn't actually realize that dynamic was at play (at an organizational level, anyway, I'm well aware individuals game # of contributions). These organizations pushing trivial/garbage patches and wasting maintainers' time absolutely have to be called out by the community as unacceptable, so I'm glad to see the post. This is a massive waste of time, time which open source contributors/maintainers often are already short on.

cjohansson 5y ago

Sounds like the typical issue of business intelligence using quantity over quality to rate work performance

croes 5y ago

"When a measure becomes a target, it ceases to be a good measure."

thih9 5y ago
Corollary: "Businesses that base their growth on A/B testing only, turn into gambling or pornography".
- Seirdy 5y ago
  Relevant xkcd: "Video Content" https://xkcd.com/1804/ https://xkcd.com/1804/ A/B and telemetry-worship reminds me of something I posted on Fedi earlier. Pasting it below: --- "Guys, I have an idea. Let's make a browser optimized for the kind of people who don't opt out of telemetry, and market it to privacy-conscious people." -- Top minds at Mozilla Corp If you market to privacy-conscious people and you base all your decisions on data from telemetry, you're gonna get a very skewed perspective. You'll see disproportionate representation from fans (people who want to enable telemetry because they love Mozilla and want to help, and will rationalize any decision Mozilla makes) and very non-technical users who don't know what telemetry is and how opt-in/out works. Technical users who disable stuff like telemetry, analytics, etc. are going to look like a rounding error. This explains a lot of Firefox's changes.
  - AwaAwa 5y ago
    This is enlightening and depressing. It's sad to see the Fox let itself go.
squabble 5y ago
I sometimes wonder if something like this has happened with money.
- deleted 5y ago
  [deleted]