4 ms·

Maintainers need their own metrics that are effective, and publish them and work to have partners use them.

duxup 5y ago
I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law. This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.
- newsclues 5y ago
  I’m not knowledgeable enough to define what good work looks like but, simply, whenever maintainers see good work, give that user a golden star. People will work for the carrot, so give carrots for the work maintainers want to see.
  - PixelOfDeath 5y ago
    Then you get them into trying to get their own people into maintainer positions to let the golden stars flow. Maybe also be more hesitant to give out this starts to the "wrong" people from other companies. Not like there already is enough politic bullshit in open source.
lmilcin 5y ago
The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue. But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.
- newsclues 5y ago
  Yes, so either accept the status quo or create positive change. Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.
  - trasz 5y ago
    What for? Metrics for metrics sake are a waste of time. The solution is to avoid wasting time on metrics, which is precisely what happened here.
  - ori_b 5y ago
    Sometimes, throwing trash into the trash bin is positive change. You don't need to put new trash in as a replacement.
- qsmlfkqjsd 5y ago
  > The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.
  - lmilcin 5y ago
    It is not about any single patch, you can't assess value of any patch objectively. For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions. In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?
    - mltdbrn 5y ago
      > For starters there isn't even an objectively established definition of value. That's doesn't mean that one cannot come up with a definition . I believe that for a long time there wasn't a proper definition for energy. > You can't objectively measure something that only has subjective definitions. But it's your claim that it can only be subjective. Somehow you don't seem able to demonstrate it, nor even show an example of a patch - or a patch set - for which no meaningful objective quality measurement could be done. Anyway, this might very well be a red herring: it doesn't matter if it's objective or not, as long as those involved agree with it. Apparently, the current measure is this KPI thing (which by the way, is very objective, since it doesn't involve the judgement of anyone), and this mail we're discussing seems to indicate its inadequacy (because it doesn't measure the right thing). Now, whether having such measure is a good thing or not is of course a different matter, and perhaps it should be discussed first.
  - spfzero 5y ago
    It might be feasible to assess a single contribution, but not feasible to assess all of them. To make this KPI work, maintainers would have to publish an objective quality rating for each accepted commit. That's an enormous amount of work, and impossible to get right anyway since the value of a change might not be immediately obvious, or might be negated by a future change, etc. The advice to companies using this KPI would be: let your own dev managers evaluate the quality of the commits of their engineers, and rate them on quality as well as quantity, then submit. Engineers will quickly stop creating busywork when their managers get wise to it.
tgsovlerkhgsel 5y ago
Doesn't work. "When a measure becomes a target, it ceases to be a good measure." You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.
- acover 5y ago
  Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it. Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.
  - simonh 5y ago
    Because that would be a huge time sink for the maintainers as well. They’re the only ones qualified to evaluate quality and they have better things to do with the time it would take. This problem is created by these companies and it’s their problem to solve.
  - jart 5y ago
    I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.
    - sdesol 5y ago
      > I tend to look on commit count as a negative when reviewing open source projects. I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment. Take the following for example: https://public-001.gitsense.com/insights/github/repos?p=impact&r=github%3Amicrosoft%2Fvscode&v=repo https://public-001.gitsense.com/insights/github/repos?p=impa... by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace. Software metrics in general isn't the issue, it's the lack of context around them that is the issue. Disclaimer: I'm the creator of the tool that I'm linking to

I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law. This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.

newsclues 5y ago
I’m not knowledgeable enough to define what good work looks like but, simply, whenever maintainers see good work, give that user a golden star. People will work for the carrot, so give carrots for the work maintainers want to see.
- PixelOfDeath 5y ago
  Then you get them into trying to get their own people into maintainer positions to let the golden stars flow. Maybe also be more hesitant to give out this starts to the "wrong" people from other companies. Not like there already is enough politic bullshit in open source.

lmilcin 5y ago

The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue. But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.

newsclues 5y ago
Yes, so either accept the status quo or create positive change. Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.
- trasz 5y ago
  What for? Metrics for metrics sake are a waste of time. The solution is to avoid wasting time on metrics, which is precisely what happened here.
- ori_b 5y ago
  Sometimes, throwing trash into the trash bin is positive change. You don't need to put new trash in as a replacement.
qsmlfkqjsd 5y ago
> The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.
- lmilcin 5y ago
  It is not about any single patch, you can't assess value of any patch objectively. For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions. In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?
  - mltdbrn 5y ago
    > For starters there isn't even an objectively established definition of value. That's doesn't mean that one cannot come up with a definition . I believe that for a long time there wasn't a proper definition for energy. > You can't objectively measure something that only has subjective definitions. But it's your claim that it can only be subjective. Somehow you don't seem able to demonstrate it, nor even show an example of a patch - or a patch set - for which no meaningful objective quality measurement could be done. Anyway, this might very well be a red herring: it doesn't matter if it's objective or not, as long as those involved agree with it. Apparently, the current measure is this KPI thing (which by the way, is very objective, since it doesn't involve the judgement of anyone), and this mail we're discussing seems to indicate its inadequacy (because it doesn't measure the right thing). Now, whether having such measure is a good thing or not is of course a different matter, and perhaps it should be discussed first.
- spfzero 5y ago
  It might be feasible to assess a single contribution, but not feasible to assess all of them. To make this KPI work, maintainers would have to publish an objective quality rating for each accepted commit. That's an enormous amount of work, and impossible to get right anyway since the value of a change might not be immediately obvious, or might be negated by a future change, etc. The advice to companies using this KPI would be: let your own dev managers evaluate the quality of the commits of their engineers, and rate them on quality as well as quantity, then submit. Engineers will quickly stop creating busywork when their managers get wise to it.

tgsovlerkhgsel 5y ago

Doesn't work. "When a measure becomes a target, it ceases to be a good measure." You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.

acover 5y ago
Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it. Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.
- simonh 5y ago
  Because that would be a huge time sink for the maintainers as well. They’re the only ones qualified to evaluate quality and they have better things to do with the time it would take. This problem is created by these companies and it’s their problem to solve.
- jart 5y ago
  I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.
  - sdesol 5y ago
    > I tend to look on commit count as a negative when reviewing open source projects. I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment. Take the following for example: https://public-001.gitsense.com/insights/github/repos?p=impact&r=github%3Amicrosoft%2Fvscode&v=repo https://public-001.gitsense.com/insights/github/repos?p=impa... by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace. Software metrics in general isn't the issue, it's the lack of context around them that is the issue. Disclaimer: I'm the creator of the tool that I'm linking to