17 ms·

Worst Software Metrics

82 points by wbharding 7y ago

smadge 7y ago
Why not let developers justify their work and point out its positive impact themselves? It would require some overhead, but they could highlight the appropriate metric if it can be quantified and also justify more qualitative improvements.
makecheck 7y ago
There’s a tendency for “metrics” to evolve into a process where the police give you speeding tickets based on the number of times you opened the passenger-side windows of your car. Oh, and if you get a flat tire it must mean that you should lose your license. Also, beware of pretty graph summaries that are presented without a healthy list of caveats and follow-ups and further explanations. Beware if no one with management power is asking the follow-up questions that matter.
lazyant 7y ago
Author concludes (before promoting his product) "The metric would consider "issues resolved," but only after normalizing for implementation complexity", so story points then?
ljw1001 7y ago
The 'best' software productivity metric I've ever seen is one I created and tested on a large code base. It only measures coding, but it's better than the other metrics for that. It is based on information theory and credits code removed equally to code added. In the end, I reported the results to the devs and then stopped doing it because there was too much opportunity for abuse by my superiors. Once they got wind of it, they wanted to know everyone's scores and I could see where it was heading. The metric: It was based on a very powerful (and slow) compression algorithm such that it could measure the change in code. Since it used compression, whitespace, boiler plate, etc had essentially no impact. It was carefully tested by comparing the results against human assessments. The only clear issue was that it did over-credit mass changes (renaming a root package in a java project which caused all files to be changed). This could probably have been handled by adjusting the scores. After looking at other metrics (bugs reported, comments on bug reports, code reviews completed, comments in code reviews, etc) it was clear that some people made _huge_ contributions through activities that were ignored by the algorithm, and some people who had great software productivity scores contributed little in these other activities. They were putting all their time into their own code. FWIW, In the same exercise, I came up with another good metric to identify problematic code (rather than problematic people). That metric used machine learning to find code that needed to be refactored. But that's a story for another time.
sbellware 7y ago
The only telltale metric of software work that I know is the ratio of time spent making progress to time spent not being able to make progress because of past mistakes made and shortcuts taken. It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer. If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get. We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement). Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science. We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't. There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.
toolslive 7y ago
Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)
- Benjammer 7y ago
  You're discussing it as a negative metric/limiting factor/base constant type of thing, but that's not the context in the article: >Thus, the "most valuable developer" as measured by LoC tends to be whoever is adding the most CSS, whitespace, and third-party libraries. It's still as terrible as ever to positively correlate developer productivity/quality/skill with LOC.
  - taeric 7y ago
    Of course, that developer almost certainly introduced the most bugs, as well. So, maybe? :) Though, agreed you are almost certainly better pulling back up and asking more product faced questions. The most prolific author, as an example, produced more sentences read by someone. Not just more sentences.
- Kinrany 7y ago
  And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.
  - wbharding 7y ago
    While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful * In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving * In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.
eirikma 7y ago
Does anybody know of a study documenting that measuring and reporting software code quality metrics actually improve the project/product team's ability to deliver? So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybody measured if these measurements actually are helpfull?
- commandlinefan 7y ago
  Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."
- wbharding 7y ago
  This is a terrific question, and a topic I've thought about quite a bit while we've built our product (context: as the OP, I've toiled on "how to prove metrics are signal-bearing" for a couple years now). The problem we always end up at is "what measurable output can we rely upon to tell the story that a project's fortunes are improving"? To date, its been challenging to find a compelling answer. This may be why I'm unaware of any study that addresses the question. One possible workaround would be to take experts in the field, and try to translate their qualitative interpretation into input that drives a metric. e.g., if Andrew Clark from React could list three directories that harbor disproportionate tech debt in their project, we could calibrate our Open Repo Directory Browser to identify the factors that were unique to that directory. This could then help managers pick out when and where its appropriate to make payments to reduce their tech debt. Unfortunately, the prominent open source developers aren't answering my emails at this point. And even if they did, it's still a long road to translate qualitative expert opinion into provable long-term benefits. But I think the environment for making that leap is as ripe as ever. We just need to keep working toward figuring out the most desirable+measurable outputs.
  - namibj 7y ago
    This sounds like something Felix von Leitner might be interested in. Consider collaborating. https://blog.fefe.de https://blog.fefe.de
commandlinefan 7y ago
> The longest has taken two weeks and is still ongoing. Even worse, the typical braindead “solution” to this problem is to insist that everything be broken up into chunks of a few hours or less each before work can begin. The end result is “if it can’t be done in a couple of hours, it isn’t worth doing”.
- perl4ever 7y ago
  It seems like I have read where "esteemed members of the community" advocate this, at least to enable estimation. Joel-on-software, maybe?
ilaksh 7y ago
I have a solution to these metric gotchas. Don't install Jira at all and then you won't be tempted by any of them. Better yet, don't hire a manager at all. Instead get one or two senior devs and include them in some business discussions and incentives.
- abathur 7y ago
  I wasn't going to say this quite the same way, but I think I was going to say roughly the same thing. The metrics are all an effort to abstract developer output into something a manager with no/minimal domain experience can understand. It feels very similar to the measurement/quantification problems distorting academic output.
- shakezula 7y ago
  This is, in my opinion, the actual best approach. Whenever I see a manager reaching for a "code metric", my immediate thought is that they're so far removed from the actual engineering of the product that they don't feel like they have any feedback so they want to reach for something to feel like they do.
- overgard 7y ago
  gasp include senior devs in a conversation about development? Blasphemy. At so many places it's like there's some unwritten law that you can't include developers in any sort of business conversation, you need to hire a project manager as a go-between in order to badly translate things between business and development so neither side actually knows what's going on.
  - mnowicki 7y ago
    Why is this so true. I usually assume that when a large percentage of people do something that seems dumb, it's usually because there is some reason I don't understand and not because they all coincidentally decided to be idiots in the same way. Occasionally there is a common pitfall that does cause them all to be dumb in the same way, and more often there's a good reason. In this case though, it's so common but I can't see how there could possibly be a reason for this. Maybe tech companies just tend to have really small meeting rooms with only a few chairs in them.
sebastianconcpt 7y ago
Yep! Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether the data is any good. When you place trust in a metric like commit count, you're liable to end off worse than if you'd used no data at all (i.e., by pitting the team against one another). Our opinion is that a less bad metric would be one that harnessed the signal lurking deep within lines of code, while rinsing away its noise. The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.
- TuringNYC 7y ago
  >> The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets. Only if you are not being rewarded for solving your own bugs. Otherwise, I've seen this go horribly awry. I was at a Big 5 consulting company where one project was big on "how well you managed issues and brought them to resolution". That meant that good developers who prioritized code coverage and mostly bug-free code did not have good issue resolution case studies to cite during promotion cycles. After a year of good developers getting dinged on this, some good developers banded together to purposefully leave known bugs in the release (especially bugs that could be blamed on vague specs -- so instead of trying and resolve issues during dev cycles, they were propagated to production). Then, the developer would swoop in and do lots of "investigation", produce a fancy write-up and email (carefully cc'ing senior management) and throw the business analyst under the bus. Senior Management naturally was impressed and some of the developers got promoted for their "leadership" and management of "critical bugs." Bad Incentives --> Bad Outcomes
  - coldcode 7y ago
    This is known as the Cobra Effect. I worked at a healthcare company where the chief architect was considered a genius by upper management because he would dive in and "save" production; in reality he was the cause of all the bugs that broke it but he did not allow anyone to fix the problems.
fake-name 7y ago
Actual URL: https://public.amplenote.com/embed/53SmZouWw5nNyBaJNEe6CSdF?hostname=www.gitclear.com https://public.amplenote.com/embed/53SmZouWw5nNyBaJNEe6CSdF?... I completely don't get this weird iframe for the entire article thing.
forgotmylogin2 7y ago
Is this operating under the assumption that there is no code review? With code review, lines of code and commit count become much harder to game. If you try to submit changes with unnecessary whitespace, unnecessary code constructs, or incomplete code, your reviewer will see it and tell you to fix it. Ultimately, trying to increase your lines of code or commit count will come back to haunt you as you spend much more time addressing comments from code review.
- majormajor 7y ago
  The sort of code review that ends up at "you can't land this until it looks exactly like the code I'd write" can be its own sort of hell...
- overgard 7y ago
  It's still super-easy to game. Just prioritize all the low-hanging fruit. Got some tickets for increasing the font size or replacing the copy in the privacy policy? Why not do those instead of working on the important but hard to debug/estimate problem of some database transactions being slow, but only sometimes?
shakezula 7y ago
At a previous job of mine, they implemented a "personal improvement plan" for employees who were below management's "desired productivity". There was no warning, no heads up that they would be measuring our GitHub accounts, and it had absolutely no context of what each person had been working on. These reviews came after a 360 peer review, and absolutely blindsided the team. Several of my coworkers left early for the day and didn't come in for the rest of the week. I got calls all day from various people asking what happened in my 1:1, what was said, etc... I wish I could show management how negatively and immediately that move affected the culture & mood of engineering at that company. It was an overnight change, and going forward it basically killed all trust in management. I stayed around for a little bit longer, but I should have quit the second they started those measurements. It absolutely murdered the confidence of our junior devs. Of the senior engineers who were put on it, none of them really had any idea what they were doing wrong or how to improve, as no feedback was given about any of that, just that they weren't enough. Some of our seniors had 15+ years of experience in software and were industry experts, but it didn't matter: They hadn't been hitting the number of commits and pull requests and tickets per week that management insisted on. When management was confronted about it, they admitted that code metrics aren't a good measurement of productivity, but then insisted that they "needed something". It turns out, they only used code metrics as a bludgeon to get rid of the employees they wanted gone, and since then that's all I've been able to see code metrics as. They're a battering weapon used to give HR a performance reason to justify firing you.
- actsasbuffoon 7y ago
  I’ve been a professional programmer for a while and worked at a variety of companies. When the teams I’ve been on decided to implement tons of metrics, it’s usually because we had tons of code quality problems. This was usually because we had inexperienced or inattentive programmers wreaking havoc. I currently work at a company where all the devs are pretty good and the code is decent. We don’t really need much in the way of metrics. If anything, we have more metrics than we should. I trust my current team to do the right thing more than I trust a linter.
- gtirloni 7y ago
  I'm sorry this experience left you scared of metrics but it doesn't mean they are bad, just that people need to know how to use them. I've had this exact conversation before and the extremism of "metrics are always weapons" was a contributing factor for me to leave a, say, less metrics-inclined workplace. Just saying there has to be balance.
chiefalchemist 7y ago
Not to get too far off topic but is anyone familiar with how Boeing runs their shop? What's management/leadership like? What metrics they use?
notmyfuture 7y ago
I’m a manager of ~25 developers (who report through a few leads). I also come from a dev background. IMO metrics are dangerous, but can also be useful. Often I find basic code metrics useful as smell tests, but would never use them alone, wouldn’t set them as goals, and wouldn’t use them alone to rank performance in a team. If someone has a very low number of commits relative to others on their team, I’ll investigate deeper - review a few Pull Requests, check if they’re working on something outside source control etc. sometimes I find problems, other times not. I’m curious what people think of this approach?
- lazyant 7y ago
  I agree, like many tests (IQ for ex) it's only useful to identify the outliers and investigate further.
- _Marak_ 7y ago
  I would think this is a common approach. I use the same approach. Scales up to hundreds of developers. I do generally try to enforce the goal that on any working day there should be at least one commit from the developer. If I notice a developer is consistently missing this metric it usually indicates trouble in the project they are working on or that the developer is just not delivering enough.
overgard 7y ago
Ugh the "velocity" one really hits home. A couple of jobs ago one of our project managers was constantly showing burn down charts (which, of course, never burned down); along with our "velocity". The funny thing is, because we were doing t-shirt-size/fibonacci, our velocity numbers varied so wildly that it was essentially a random number generator. Then we were required fit all our stories within a "budget" of our "average" velocity.. What are you supposed to do with the average of a random number between 0 and 200? Apparently a lot, because these insane numbers that had no actual tie to reality were used to determine how much we were allowed to "commit" to. Never mind that some months we'd finish 50 story points and some months 250 (because, again, the numbers were a fiction); we were always trying to fit things into 90 story points. Luckily all this useless theater only took up an entire day every two weeks. If you assume everyone on the team was making about $50 an hour, and it was about 6-8 people in attendance to these things, then it came out that this very dumb "estimation" process took 6 people * $50/hr * 6 hours... so about 1800 dollars. (Not including the project managers salary, which given that he drove around a very fancy luxury car, was probably a lot more than the rest of us). And people wonder why I don't take scrum seriously.
- winrid 7y ago
  To be fair, that's very poorly implemented. You should have reference stories for what is a 1 in complexity and the highest value should be low, like 13, which forces you to break things down and forces the PM to write better specs.
  - overgard 7y ago
    We did all of these things. But we had this ridiculous Fibonacci thing where you would end up with a conversation like this: dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6. pm: no 6 is not allowed, it must be either a 5 or an 8. dev: ok, lets call it an 8. Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on this story 20% off, just to satisfy a rule. I'm perfectly fine with that, if we're only talking about that one story. If we're both happy that this might go 20% faster or 20% slower than predicted, no big deal. But if you're going to include it in a larger rollup, you've included a very large error term with your number that you need to carry along with the rest of the calculations. Did you ever take a physics course in college? (A real one -- not one of the baby ones they gave to business and liberal art majors). If they were good, they probably made you deal with carrying an error term in your math. You probably even hated it because it was a pain in the ass to deal with, but it's also kind of important. If you're adding together 50 numbers that are all off by 20%-50% and chucking away the error term, your summary and conclusion are absolute garbage if you do not properly keep the error term included. Math is unforgiving like that. Presenting that absolute garbage as "data" is, in my opinion, professional incompetence. It just is. I guess the softening factor is that so many people do it that it's not an individual incompetence, it's a lack of competence in the profession itself essentially.
jknoepfler 7y ago
In order to add value to a dev team as a manager, one must have an intimate understanding of the work developers do. There isn't any value in abstracting away from that work when evaluating the contribution of team members. "Context, action, outcome" is a sufficiently standardized formalism for evaluating performance (assuming outcomes are measured in terms of business value). Accurately reporting context, action, outcome requires understanding both what devs do, why they do it, and how it ultimately produces business value. There's no shortcut. If your management can't meaningfully do that because they are non technical or managing too many people, or whatever, then they are actively harming your team and should be removed with prejudice. There is no reasonable way to conclude that devs can be effectively measured with garbage stats. Even suggesting it seriously indicates incompetence or laziness.
zylepe 7y ago
Best software metric: lines of code removed
vageli 7y ago
Is the measure of an architect the number of lines in a blueprint? Is the measure of a craftsman the number of finished goods produced? It seems to me that different projects in the same domain should use different attributes to evaluate success. Why do we attempt to use uniform measures across-the-board? Is it lazy management or does delving deep into project-specific metrics make it hard for large enterprises to compare employees in similar functions? Or something else?

82 points by wbharding 7y ago

Why not let developers justify their work and point out its positive impact themselves? It would require some overhead, but they could highlight the appropriate metric if it can be quantified and also justify more qualitative improvements.

makecheck 7y ago

There’s a tendency for “metrics” to evolve into a process where the police give you speeding tickets based on the number of times you opened the passenger-side windows of your car. Oh, and if you get a flat tire it must mean that you should lose your license. Also, beware of pretty graph summaries that are presented without a healthy list of caveats and follow-ups and further explanations. Beware if no one with management power is asking the follow-up questions that matter.

lazyant 7y ago

Author concludes (before promoting his product) "The metric would consider "issues resolved," but only after normalizing for implementation complexity", so story points then?

ljw1001 7y ago

The 'best' software productivity metric I've ever seen is one I created and tested on a large code base. It only measures coding, but it's better than the other metrics for that. It is based on information theory and credits code removed equally to code added. In the end, I reported the results to the devs and then stopped doing it because there was too much opportunity for abuse by my superiors. Once they got wind of it, they wanted to know everyone's scores and I could see where it was heading. The metric: It was based on a very powerful (and slow) compression algorithm such that it could measure the change in code. Since it used compression, whitespace, boiler plate, etc had essentially no impact. It was carefully tested by comparing the results against human assessments. The only clear issue was that it did over-credit mass changes (renaming a root package in a java project which caused all files to be changed). This could probably have been handled by adjusting the scores. After looking at other metrics (bugs reported, comments on bug reports, code reviews completed, comments in code reviews, etc) it was clear that some people made _huge_ contributions through activities that were ignored by the algorithm, and some people who had great software productivity scores contributed little in these other activities. They were putting all their time into their own code. FWIW, In the same exercise, I came up with another good metric to identify problematic code (rather than problematic people). That metric used machine learning to find code that needed to be refactored. But that's a story for another time.

sbellware 7y ago

The only telltale metric of software work that I know is the ratio of time spent making progress to time spent not being able to make progress because of past mistakes made and shortcuts taken. It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer. If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get. We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement). Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science. We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't. There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.

toolslive 7y ago

Actually, LOC isn't that bad. Studies have shown - that complexity metrics are highly correlated with LOC. - people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind. It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more. Both facts are discussed at length in "Making Software" (Oram, Wilson)

Benjammer 7y ago
You're discussing it as a negative metric/limiting factor/base constant type of thing, but that's not the context in the article: >Thus, the "most valuable developer" as measured by LoC tends to be whoever is adding the most CSS, whitespace, and third-party libraries. It's still as terrible as ever to positively correlate developer productivity/quality/skill with LOC.
- taeric 7y ago
  Of course, that developer almost certainly introduced the most bugs, as well. So, maybe? :) Though, agreed you are almost certainly better pulling back up and asking more product faced questions. The most prolific author, as an example, produced more sentences read by someone. Not just more sentences.
Kinrany 7y ago
And at the same time you still can't use LoC to measure productivity and reward for it because of the Goodhart's law.
- wbharding 7y ago
  While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful * In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving * In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.

eirikma 7y ago

Does anybody know of a study documenting that measuring and reporting software code quality metrics actually improve the project/product team's ability to deliver? So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybody measured if these measurements actually are helpfull?

commandlinefan 7y ago
Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."
wbharding 7y ago
This is a terrific question, and a topic I've thought about quite a bit while we've built our product (context: as the OP, I've toiled on "how to prove metrics are signal-bearing" for a couple years now). The problem we always end up at is "what measurable output can we rely upon to tell the story that a project's fortunes are improving"? To date, its been challenging to find a compelling answer. This may be why I'm unaware of any study that addresses the question. One possible workaround would be to take experts in the field, and try to translate their qualitative interpretation into input that drives a metric. e.g., if Andrew Clark from React could list three directories that harbor disproportionate tech debt in their project, we could calibrate our Open Repo Directory Browser to identify the factors that were unique to that directory. This could then help managers pick out when and where its appropriate to make payments to reduce their tech debt. Unfortunately, the prominent open source developers aren't answering my emails at this point. And even if they did, it's still a long road to translate qualitative expert opinion into provable long-term benefits. But I think the environment for making that leap is as ripe as ever. We just need to keep working toward figuring out the most desirable+measurable outputs.
- namibj 7y ago
  This sounds like something Felix von Leitner might be interested in. Consider collaborating. https://blog.fefe.de https://blog.fefe.de

commandlinefan 7y ago

> The longest has taken two weeks and is still ongoing. Even worse, the typical braindead “solution” to this problem is to insist that everything be broken up into chunks of a few hours or less each before work can begin. The end result is “if it can’t be done in a couple of hours, it isn’t worth doing”.

perl4ever 7y ago
It seems like I have read where "esteemed members of the community" advocate this, at least to enable estimation. Joel-on-software, maybe?

ilaksh 7y ago

I have a solution to these metric gotchas. Don't install Jira at all and then you won't be tempted by any of them. Better yet, don't hire a manager at all. Instead get one or two senior devs and include them in some business discussions and incentives.

abathur 7y ago
I wasn't going to say this quite the same way, but I think I was going to say roughly the same thing. The metrics are all an effort to abstract developer output into something a manager with no/minimal domain experience can understand. It feels very similar to the measurement/quantification problems distorting academic output.
shakezula 7y ago
This is, in my opinion, the actual best approach. Whenever I see a manager reaching for a "code metric", my immediate thought is that they're so far removed from the actual engineering of the product that they don't feel like they have any feedback so they want to reach for something to feel like they do.
overgard 7y ago
gasp include senior devs in a conversation about development? Blasphemy. At so many places it's like there's some unwritten law that you can't include developers in any sort of business conversation, you need to hire a project manager as a go-between in order to badly translate things between business and development so neither side actually knows what's going on.
- mnowicki 7y ago
  Why is this so true. I usually assume that when a large percentage of people do something that seems dumb, it's usually because there is some reason I don't understand and not because they all coincidentally decided to be idiots in the same way. Occasionally there is a common pitfall that does cause them all to be dumb in the same way, and more often there's a good reason. In this case though, it's so common but I can't see how there could possibly be a reason for this. Maybe tech companies just tend to have really small meeting rooms with only a few chairs in them.

sebastianconcpt 7y ago

Yep! Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether the data is any good. When you place trust in a metric like commit count, you're liable to end off worse than if you'd used no data at all (i.e., by pitting the team against one another). Our opinion is that a less bad metric would be one that harnessed the signal lurking deep within lines of code, while rinsing away its noise. The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.

TuringNYC 7y ago
>> The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets. Only if you are not being rewarded for solving your own bugs. Otherwise, I've seen this go horribly awry. I was at a Big 5 consulting company where one project was big on "how well you managed issues and brought them to resolution". That meant that good developers who prioritized code coverage and mostly bug-free code did not have good issue resolution case studies to cite during promotion cycles. After a year of good developers getting dinged on this, some good developers banded together to purposefully leave known bugs in the release (especially bugs that could be blamed on vague specs -- so instead of trying and resolve issues during dev cycles, they were propagated to production). Then, the developer would swoop in and do lots of "investigation", produce a fancy write-up and email (carefully cc'ing senior management) and throw the business analyst under the bus. Senior Management naturally was impressed and some of the developers got promoted for their "leadership" and management of "critical bugs." Bad Incentives --> Bad Outcomes
- coldcode 7y ago
  This is known as the Cobra Effect. I worked at a healthcare company where the chief architect was considered a genius by upper management because he would dive in and "save" production; in reality he was the cause of all the bugs that broke it but he did not allow anyone to fix the problems.

fake-name 7y ago

Actual URL: https://public.amplenote.com/embed/53SmZouWw5nNyBaJNEe6CSdF?hostname=www.gitclear.com https://public.amplenote.com/embed/53SmZouWw5nNyBaJNEe6CSdF?... I completely don't get this weird iframe for the entire article thing.

forgotmylogin2 7y ago

Is this operating under the assumption that there is no code review? With code review, lines of code and commit count become much harder to game. If you try to submit changes with unnecessary whitespace, unnecessary code constructs, or incomplete code, your reviewer will see it and tell you to fix it. Ultimately, trying to increase your lines of code or commit count will come back to haunt you as you spend much more time addressing comments from code review.

majormajor 7y ago
The sort of code review that ends up at "you can't land this until it looks exactly like the code I'd write" can be its own sort of hell...
overgard 7y ago
It's still super-easy to game. Just prioritize all the low-hanging fruit. Got some tickets for increasing the font size or replacing the copy in the privacy policy? Why not do those instead of working on the important but hard to debug/estimate problem of some database transactions being slow, but only sometimes?

shakezula 7y ago

At a previous job of mine, they implemented a "personal improvement plan" for employees who were below management's "desired productivity". There was no warning, no heads up that they would be measuring our GitHub accounts, and it had absolutely no context of what each person had been working on. These reviews came after a 360 peer review, and absolutely blindsided the team. Several of my coworkers left early for the day and didn't come in for the rest of the week. I got calls all day from various people asking what happened in my 1:1, what was said, etc... I wish I could show management how negatively and immediately that move affected the culture & mood of engineering at that company. It was an overnight change, and going forward it basically killed all trust in management. I stayed around for a little bit longer, but I should have quit the second they started those measurements. It absolutely murdered the confidence of our junior devs. Of the senior engineers who were put on it, none of them really had any idea what they were doing wrong or how to improve, as no feedback was given about any of that, just that they weren't enough. Some of our seniors had 15+ years of experience in software and were industry experts, but it didn't matter: They hadn't been hitting the number of commits and pull requests and tickets per week that management insisted on. When management was confronted about it, they admitted that code metrics aren't a good measurement of productivity, but then insisted that they "needed something". It turns out, they only used code metrics as a bludgeon to get rid of the employees they wanted gone, and since then that's all I've been able to see code metrics as. They're a battering weapon used to give HR a performance reason to justify firing you.

actsasbuffoon 7y ago
I’ve been a professional programmer for a while and worked at a variety of companies. When the teams I’ve been on decided to implement tons of metrics, it’s usually because we had tons of code quality problems. This was usually because we had inexperienced or inattentive programmers wreaking havoc. I currently work at a company where all the devs are pretty good and the code is decent. We don’t really need much in the way of metrics. If anything, we have more metrics than we should. I trust my current team to do the right thing more than I trust a linter.
gtirloni 7y ago
I'm sorry this experience left you scared of metrics but it doesn't mean they are bad, just that people need to know how to use them. I've had this exact conversation before and the extremism of "metrics are always weapons" was a contributing factor for me to leave a, say, less metrics-inclined workplace. Just saying there has to be balance.

chiefalchemist 7y ago

Not to get too far off topic but is anyone familiar with how Boeing runs their shop? What's management/leadership like? What metrics they use?

notmyfuture 7y ago

I’m a manager of ~25 developers (who report through a few leads). I also come from a dev background. IMO metrics are dangerous, but can also be useful. Often I find basic code metrics useful as smell tests, but would never use them alone, wouldn’t set them as goals, and wouldn’t use them alone to rank performance in a team. If someone has a very low number of commits relative to others on their team, I’ll investigate deeper - review a few Pull Requests, check if they’re working on something outside source control etc. sometimes I find problems, other times not. I’m curious what people think of this approach?

lazyant 7y ago
I agree, like many tests (IQ for ex) it's only useful to identify the outliers and investigate further.
_Marak_ 7y ago
I would think this is a common approach. I use the same approach. Scales up to hundreds of developers. I do generally try to enforce the goal that on any working day there should be at least one commit from the developer. If I notice a developer is consistently missing this metric it usually indicates trouble in the project they are working on or that the developer is just not delivering enough.

overgard 7y ago

Ugh the "velocity" one really hits home. A couple of jobs ago one of our project managers was constantly showing burn down charts (which, of course, never burned down); along with our "velocity". The funny thing is, because we were doing t-shirt-size/fibonacci, our velocity numbers varied so wildly that it was essentially a random number generator. Then we were required fit all our stories within a "budget" of our "average" velocity.. What are you supposed to do with the average of a random number between 0 and 200? Apparently a lot, because these insane numbers that had no actual tie to reality were used to determine how much we were allowed to "commit" to. Never mind that some months we'd finish 50 story points and some months 250 (because, again, the numbers were a fiction); we were always trying to fit things into 90 story points. Luckily all this useless theater only took up an entire day every two weeks. If you assume everyone on the team was making about $50 an hour, and it was about 6-8 people in attendance to these things, then it came out that this very dumb "estimation" process took 6 people * $50/hr * 6 hours... so about 1800 dollars. (Not including the project managers salary, which given that he drove around a very fancy luxury car, was probably a lot more than the rest of us). And people wonder why I don't take scrum seriously.

winrid 7y ago
To be fair, that's very poorly implemented. You should have reference stories for what is a 1 in complexity and the highest value should be low, like 13, which forces you to break things down and forces the PM to write better specs.
- overgard 7y ago
  We did all of these things. But we had this ridiculous Fibonacci thing where you would end up with a conversation like this: dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6. pm: no 6 is not allowed, it must be either a 5 or an 8. dev: ok, lets call it an 8. Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on this story 20% off, just to satisfy a rule. I'm perfectly fine with that, if we're only talking about that one story. If we're both happy that this might go 20% faster or 20% slower than predicted, no big deal. But if you're going to include it in a larger rollup, you've included a very large error term with your number that you need to carry along with the rest of the calculations. Did you ever take a physics course in college? (A real one -- not one of the baby ones they gave to business and liberal art majors). If they were good, they probably made you deal with carrying an error term in your math. You probably even hated it because it was a pain in the ass to deal with, but it's also kind of important. If you're adding together 50 numbers that are all off by 20%-50% and chucking away the error term, your summary and conclusion are absolute garbage if you do not properly keep the error term included. Math is unforgiving like that. Presenting that absolute garbage as "data" is, in my opinion, professional incompetence. It just is. I guess the softening factor is that so many people do it that it's not an individual incompetence, it's a lack of competence in the profession itself essentially.

jknoepfler 7y ago

In order to add value to a dev team as a manager, one must have an intimate understanding of the work developers do. There isn't any value in abstracting away from that work when evaluating the contribution of team members. "Context, action, outcome" is a sufficiently standardized formalism for evaluating performance (assuming outcomes are measured in terms of business value). Accurately reporting context, action, outcome requires understanding both what devs do, why they do it, and how it ultimately produces business value. There's no shortcut. If your management can't meaningfully do that because they are non technical or managing too many people, or whatever, then they are actively harming your team and should be removed with prejudice. There is no reasonable way to conclude that devs can be effectively measured with garbage stats. Even suggesting it seriously indicates incompetence or laziness.

zylepe 7y ago

Best software metric: lines of code removed

vageli 7y ago

Is the measure of an architect the number of lines in a blueprint? Is the measure of a craftsman the number of finished goods produced? It seems to me that different projects in the same domain should use different attributes to evaluate success. Why do we attempt to use uniform measures across-the-board? Is it lazy management or does delving deep into project-specific metrics make it hard for large enterprises to compare employees in similar functions? Or something else?