17 ms·

Can developer productivity be measured?

113 points by 97amarnathk 6y ago

LeviIsaac 6y ago
Most software projects get managed with a ticketing system that logs the work to be done as individual tickets. Counting the number of cards a developer closes over a certain period allows us to see what actual work is getting closed off. Measuring closed tickets is an excellent metric if the tasks are written well and assigned based on business priority. When more tickets get closed, more good things are happening with the project, be that bugs getting closed off or features made.
- 2rsf 6y ago
  What you are saying is that my colleague who has been working on one ticket, solving a critical bug, for the last 2 or 3 weeks is a an unproductive one since he haven't closed a ticket for a while ? Counting closed tickets is indeed a measure for something, but by itself it's far from being a good indicator.
  - lkxijlewlf 6y ago
    Yes, he's unproductive. What he should have done was broke it up into dozens of tiny pieces so that he could inflate his ticket count. It is more acceptable to the spreadsheet to have "Hard problem part 1", "Hard problem part 2", "Hard problem part ...", than it is to simply have "Hard problem" and take longer to do it.
    - duxup 6y ago
      Hard problem part 1 doesn't really indicate that they did / will solve a problem though. It's work, but is that 'productivity'? That seems kinda arbitrary to just manipulate the issue into tiny pieces to fit some sort of metrics system... but not reflective of the actual work. That seems to just lead to the typical gamification that comes with counting tickets and other metrics systems that end up being arbitrary or even easily manipulated. Ticket measuring just seems like asking for Goodhart’s Law.
      - commandlinefan 6y ago
        In my experience, project managers work as hard as they can to fight that tendency too: they usually insist that tickets be observable and testable independently, specifically to discourage developers from logging time on non-visible tasks.
      - TheCoelacanth 6y ago
        Yeah, you aren't measuring level of productivity. You are measuring level of tolerance for bullshit ticket creation.
    - commandlinefan 6y ago
      I hope you're being sarcastic.
      - TameAntelope 6y ago
        Probably, but to represent the non-sarcastic point of view, it comes down to "[person] in the room" syndrome[1]. Spending 2-3 weeks on a bug is never good, if you're not communicating progress, and one way to communicate progress is to break down the work into smaller chunks. Does it literally have to be individual JIRA tickets? No way, but going off for 2-3 weeks doesn't give the business the insight it needs to in order to wisely invest time/effort into work being executed. [1] https://medium.com/machine-words/a-guy-in-a-room-bbbe058645ef https://medium.com/machine-words/a-guy-in-a-room-bbbe058645e... (I thought Joel Spolsky said this but I can't actually find the original source, if anyone has it I'd appreciate it!)
        valenterry 6y ago
        > Spending 2-3 weeks on a bug is never good, if you're not communicating progress Assuming that it is important to fix the bug and that the developer is competent and trusted - why not? What would communicating progress improve here? Mind that you cannot communicate when it is done (otherwise it would not be a hard bug) you can only communicate what you have done so far and what you try next. But what kind of business value does that create?
        TameAntelope 6y ago
        The value is that the developer doesn’t reasonably have the context necessary to make the call whether or not, over time, the issue is worth continuing to invest time to resolve. Not that they couldn’t make the call if they had all the info, but the time required to gather and understand all the context would be a second full-time job.
        learc83 6y ago
        That's why the developer has a manager who is a human who can talk to them and find out what they are working on, instead of a robot that can only process tickets created and tickets completed.
    - sokoloff 6y ago
      You can measure progress of a military campaign by counting and reporting on “bullets fired” yet that’s not how military strategists operate.
      - byecomputer 6y ago
        On the other hand, doing busywork is a pretty integral part of being in the military
    - valenterry 6y ago
      Great. Now I can't find anything anymore in the ticket system because every ticket that was non-trivial has been broken down into a dozen of other tickets. There's only one way to call this: ticket system abuse.
  - x86amdryzen 6y ago
    Stop using some very rare edge cases as an argument. You are just showing your absolute incompetence.
  - matthewmacleod 6y ago
    The mistake there would be getting that specific about it – it's not a useful method for an individual developer, but it _is_ a useful method for the team overall, where these effects get amortised. (Although in this specific case, your colleague who has been working on one ticket for two or three weeks is operating in a way that I find is usually pretty harmful for productivity overall. "Solving a critical bug" is almost universally something that can be broken down further.)
- exlurker 6y ago
  Nice, I call dibs on the low hanging fruits!
- coldcode 6y ago
  Paying for dead cobras always pays off. Also ticket != value. Lots of tickets for things that involve almost no work and things that actually make a difference to the customer/product are not equal. Everything I work on is new products/projects and tickets come in all sizes and shapes, and often change daily as some exec crams in more new ideas or some designer or product person "clarifies" the ticket, even after the work is done. Tickets are often written and estimated long before decisions are actually made. Defects are written that require a lot of investigation only to discover it's some other teams problem and you can't do anything or turns out to be a temporary service outage no one communicated or misconfiguration in some CMS or even plain simply not understanding what the product does. Measuring productivity by tickets closed is a whole pile of dead snakes.
  - jrs235 6y ago
    In case people don't understand the dead cobras comment: https://en.wikipedia.org/wiki/Cobra_effect https://en.wikipedia.org/wiki/Cobra_effect It pertains to perverse incentives.
  - TameAntelope 6y ago
    What do you think of the idea that if tickets aren't doing a good job of representing value to the customer in some form (even if it's second or third order value), those tickets are poorly written and it's a "garbage in/garbage out" situation? It's not really a helpful observation, but I'm curious if there's a way for the relationship between "people asking for things" and "people building things" to be repeatably fruitful (IMO it's very possible that reliably producing customer value is either insanely hard and/or not doable consistently).
    - coldcode 6y ago
      Some things are incredibly important but not directly related to the customer; like services called by service called by services called by clients. It's not always easy to connect the dots especially in a micro service world. But without the ultimate service the customer can't do anything. Making good tickets across a dozen organizations and 100's of people is hard to ever get right. Which is why counting tickets is sort of pointless, you might have a ticket to add a single value to a database and without it, the whole product doesn't work, but you have no idea since there are 10 layers between you and the real customer.
- zarkov99 6y ago
  I think there is some value to that metric but I would not call it excellent. A really good developer might come up with a slight requirement change or an engineering detour that makes many tickets meaningless, he might improve tooling such that things that took hours now take minutes, he might be able to write tests or come up with new processes that increase quality 10x. If we think of guys like Jeff Dean, the ability to close tickets written by project managers is not what makes them stand out.
  - commandlinefan 6y ago
    Well, don't forget you're supposed to "estimate" everything (based on a 10-second glance at the description) before you do it, and you're also measured on the accuracy of your estimates.
- danjac 6y ago
  That assumes all tickets are the same. A developer might take on a very difficult task with a lot of hidden technical complexity that ties them up for weeks. Another might pick up little bugs and small text updates. With no other insight, the metric is meaningless. It becomes easy to game by avoiding any difficult and time consuming tickets - such as refactoring - as much as possible, instead picking the quick and easy things that make your metric look good.
- 908B64B197 6y ago
  > Measuring closed tickets is an excellent metric if the tasks are written well and assigned based on business priority Don't forget the weight. I've seen single tickets taking weeks for bug investigation.
- harryf 6y ago
  Sorry but I strongly disagree. In fact I’ll come out and say its perhaps one of the worst ways you can measure productivity. At best you’re measuring _activity_ not productivity. You just turned a group of smart people into headless chickens jumping on whatever ticket so they can to look busy. Which cultivates an environment of fear, which in turn kills deep thought and creativity... two essential ingredients for good software. I could even argue that ticketing systems are the bane of good software, making real priorities intransparent... but that’s a rabbit hole I won’t go into here. Instead I’d argue we shouldn’t be trying to measure developer productivity at all. Productivity in software development is non-linear and difficult to assign individually. How do you measure the productivity of that “lazy guy” that had an amazing shower thought one morning, implemented it by lunchtime, which in turn leads to the company making millions more by the end of the year? Or what about the person on the team that spends most of their time supporting the rest of the team, unblocking them and helping them be productive? Two examples of why we shouldn’t even be trying to measure developer productivity. My own experience after 25 years in this industry is the moment someone says “but how do we measure developer productivity?” is the moment that companies software products begins a long, slow death. Ultimately what development teams and companies (not individuals) should be measured on is _results_ that positively impact customers and business. When the product is a success, no one cares about individual productivity.
BrandonMarc 6y ago
He mentions increasing salary won't lead to increased productivity ... and that's true, if the same developer remains. But what if we remove that constraint? What if increased salary means a higher quality of developer takes the position? Wouldn't this mean higher productivity? Bit of a cold scenario, but one way to game it out is hypothetically removing the current dev and then hiring someone better at double the pay. Or, less unfair seeming, double the pay by hiring a second dev. That might not double productivity ... depending on the situation it might 1.5x it, or just as easily 4x it.
- digitalsushi 6y ago
  Two engineers who are not aligned with each other's goals will easily do the work of zero people.
  - gootdude 6y ago
    Well said, this point can not be stated enough!
- kemiller2002 6y ago
  I don't completely agree with this. Money can affect developer productivity, but it takes time to realize its value. Money makes things easier. That person whose being paid the smallest amount possible, is probably trying to just scrape by. Paying people more money frees their personal life up to handle other things, less stress about family stuff etc. I never believed the "leave your personal life at the door," thing. How you live affects your work, and making life easier makes people more productive (on average, not all cases). Now this isn't an immediate thing. It's an investment and it takes time. The person has to realize that money is available (emotionally) and start to trust it will be there.
- suprfnk 6y ago
  > a higher quality of developer So what is a "higher quality of developer"?
- throwaway0950 6y ago
qz2 6y ago
Developer productivity is inversely proportional to complaining. Listen to complaints carefully.
- digitalsushi 6y ago
  I saw your text as a light grey so I decided to re-read it a few times. I absolutely agree with you. The people who complain the least should be paid the most attention when they do.
  - kemiller2002 6y ago
    There is a lecture by the late Randy Pausch (https://www.youtube.com/watch?v=ji5_MqicxSo&vl=en https://www.youtube.com/watch?v=ji5_MqicxSo&vl=en). The gist of the part I'm mentioning is, "When I stop correcting you, I've given up on you." People who don't voice their opinions aren't necessarily happy, they quite possibly have decided it's not worth trying to change things.
    - lmilcin 6y ago
      Yes, I think promoting people who DO NOT complain is problematic. There is no right way to deal with this other than to listen to complains and figure out what it is. One way of thinking about complaining is that it is a form of feedback. As a manager, you don't want to silence people giving you feedback, frankly, this is about as stupid thing as you can do. Better way to deal with complaints is to educate on what kind of complaints are productive and what kinds are destructive. For example, I try (not always succeed) to restrict to myself to only complain about things that I am ready to solve if somebody tells me "go ahead, fix it".
    - Viliam1234 6y ago
      Funny how this can reinforce the idea that only incompetent people complain. Imagine a situation where the company and/or the project have a few serious problems, but the company refuses to fix or even admit any of that. The developers who couldn't live with the problems have already quit. The developers who remained have stopped complaining, because they have given up. A new developer comes, notices the problems, and starts complaining about them. People notice that the newbie makes a fuss, but nothing changes. Later the developer either quits, or gets used to it and stops complaining. Here is how the management probably interprets the situation: "People with the least experience complain most. The correct approach is to ignore them, and wait for them to grow up. More experienced developers have realistic expectations and mature behavior."
burade 6y ago
Any decently competent technical leader can tell if a developer is being productive or not. It's stupid to waste time trying to measure something that is virtually unmeasurable.
- NortySpock 6y ago
  "It's unmeasurable, but everyone can tell." is that what you're saying? Seems like a No True Scotsman fallacy to say only good technical leaders can tell if a developer is being productive and in the same breath say it's unmeasurable. hours worked, bugs fixed, tickets closed, costs saved, clients saved, KPIs/OKRs hit, time in queue, hours-to-close-ticket, uptime, SLAs hit... surely some collection of indicators, while not a pure signal, would let you highlight outliers either above or below the curve.
  - valenterry 6y ago
    The funny thing is, except for "tickets closed" and maybe "costs saved" there is no metric here you can directly attribute to a single developer, and probably not even a team. Even "hours worked". What do you mean, hours spent in the office? How do you know the person was not drinking coffee or staring at their code in thought of something else. That being said, I think your metrics are good. And even single developers should be measured by them (which means all developers get measured by the same metric and get the same value). Why? Because it helps the business of SLAs are hit, no matter why they are hit.
  - astrobe_ 6y ago
    All of your indicators mean absolutely nothing if you do not measure the difficulty of the tasks submitted. Do that, then we can talk. It's like saying, "your productivity as a carpenter is how many houses you build in a year" while refusing to take into account how big or complex to build are those houses.
  - deleted 6y ago
    [deleted]
- rajacombinator 6y ago
jhunter1016 6y ago
Productivity is important, sure. But as with all other professions in which people interact, the interpersonal skills and behavior tend to be more important IMO. Productivity can be massively impacted (positively and negatively) by how well people communicate and get along with each other.
forbushbl 6y ago
As an individual I often wonder if my contributions are meaningful. The author says, “individual performance is best left for individual contributors to measure in themselves and each other.” How can individuals possibly measure their own performance if it can’t be measured externally?
- valenterry 6y ago
  Of course it can be measured externally, it is just too expensive.
anxiostial 6y ago
It is pretty ironic how much time is wasted on trying measure developer productivity.
- orky56 6y ago
  Why is it ironic? The stakeholders for increased developer productivity go beyond just developers. Even the slightest increase in developer productivity, let alone the ability to objectively measure it, is the holy grail of software development. Companies with access to nearly infinite resources can and would deploy them for a marginal gain in developer productivity. So much emphasis is spent on hiring the most brilliant minds and then on managing their projects and time so why not on optimizing their output.
  - anxiostial 6y ago
    You are making a lot of statements here without anything to really back it up, why is it the holy grail of software development? measuring output and actually improving developer performance are not even remotely the same thing, would it not make more sense to spend time on anything related to the actual development, like training developers for example?
ExcavateGrandMa 6y ago
measuring the copy/paste productivity is bad for the mind :Ð KrkkrkrkrKRKrkrkrkr.
GartzenDeHaes 6y ago
One of the most useful programmer metrics that I've found is code churn: (new lines + deleted lines) / total changed lines. Instead of telling you how much work your programmers are doing, this metric tells you what kind of work your programmers are doing. Small numbers mean bug fixing (end of project and maintenance) and large numbers mean new development and features.
- sixstringtheory 6y ago
  What about high deletion amounts? I've merged PRs with hundreds of thousands of lines deleted and none added. It took quite a bit of sleuthing to figure out someone had left entire copies of directories side by side with different names, where one was completely unused. Conversely, someone had a huge addition to the repo that actually was total garbage.
  - GartzenDeHaes 6y ago
    A large change to the code base would indicate to me that the product isn't ready to ship or test, possibility. At the very least, it would prompt a discussion about the code and how we are managing it. Maybe we need some process changes or tools to prevent junk code from accumulating. Edit: if you're talking about the math, I think "changed" includes added and deleted. So, it's the ratio of added and deleted to the total change.
rileymat2 6y ago
I have never felt my individual productivity go up. It feels like as I progress my individual work stays the same, but helping others eats any efficiency gains I personally make. As if when you are new to a module, you are slow because you don’t know anything, then once you have expertise, you are slow because you know everything and are helping others. Would be interesting to measure this somehow.
- orky56 6y ago
  This is the only way you can scale your time, by ramping up others to be as efficient as you are. Although you might be becoming less of a contributor individually, you are enabling the larger group. This type of productivity can definitely be tracked based on how many people you have helped and their corresponding lineage of knowledge and work output.
- Viliam1234 6y ago
  > As if when you are new to a module, you are slow because you don’t know anything, then once you have expertise, you are slow because you know everything and are helping others. This suggests that the proper way to keep team productivity high is to have all team members working on the product since the beginning, and treat them well so that they don't quit and don't have to be replaced by new ones. Maybe even start with slightly more people on the project than necessary, so if a few of them quit during the project for unrelated reasons, you can still finish the project with the remaining ones. Probably not going to happen, because this goes against maximizing short-term productivity at the beginning of the project. The short-term productivity is maximized by having the team as small as possible, and only worrying about problems after they happen.
carapace 6y ago
I once talked to a retired hardware engineer, a fellow who made real electronic devices, not software. He told me that, over the whole course of his career, 80% of the projects he worked on never made it to market. In other words, 4/5th of his total "productivity" turned out to be waste. Make of it what you will.
- BrandonMarc 6y ago
  Put differently, 80% of his time gave his brain training and practice which likely improved the 20% that made it to market. Also consider, frustrating though I'm sure that was, he probably still got paid for his effort in the 80%.
  - edelans 6y ago
    Interesting point of view, thanks for putting it that way ! It makes me think of artists : - how many hours did a musician spent on his instrument before selling his first record ? - How many drawings/paintings before Picasso could sell something ? (etc.)
- 1123581321 6y ago
  He could not have chosen to only attempt the successful projects.
  - deleted 6y ago
    [deleted]
- triceratops 6y ago
  Why didn't he just work on projects that would make it to market? /s
HourglassFR 6y ago
Maybe when developer productivity measurment becomes standard accross the industry we will realise that tech workers are in fact workers. Cogs in a machine. And not independant individuals imposing their will to the world through sheer will like some Randian hero. Maybe then it will then be plainly evident that developers are as alienated as any service worker, and in the end as disposable in the eyes of the shareholders. Will we then organize with other workers to create better working conditions for everyone or will there be fewer and fewer developers working with ever more powerful technology chasing richer than ever VCs?
- 908B64B197 6y ago
  > as disposable in the eyes of the shareholders. If a company is willing to sacrifice engineering talent and institutional knowledge for short term gains... Good luck staying in business. Reference: Every outsourcing project I've seen.
  - rusticpenn 6y ago
    They are called Best cost countries now.
- yourapostasy 6y ago
  > Maybe when developer productivity measurment becomes standard accross the industry... I suggest you look to database models/schemas standardization for an indication of how close this is coming to fruition. I personally can't measure developer productivity at a fine-grained level until requirements are stabilized, and I personally cannot stabilize requirements unless the domain is so well known the data store is standardized. I had hoped SAP would lead the charge through empirically iterating towards standards, but they left out the huge small and mid-size business markets with what they use today. And what they use today is still far from industries' standards. We're no closer to standardization than when I started in software decades ago. We don't even have standard means of storing, transforming, displaying and tracking metadata upon calendars, addresses, phone numbers, names, and lots of other ephemera I can rattle off, within a single stakeholder industry, not to speak of within the software industry in general. There have certainly been efforts to standardize like Silverston's, but they haven't caught traction. I'd sure like to see that happen, because it would short-circuit a lot of the discussions I engage with stakeholders to only the site-specific requirements, where I really add business value. Instead, I have to derive the data model from intricate discussion of their requirements, since they themselves have not agreed upon the parts that are common across their respective industries, so I end up at the start of dicussions with all sorts of little twisty pieces of a data model, all alike.
jeffbee 6y ago
Productivity as a software developer consists chiefly in not making mistakes. That can lead to the situation where your best developers may appear to do nothing for long stretches. Research and deliberation are desirable. Blind hacking is the least valuable yet most visible activity of inexperienced programmers. All common "objective" measures of productivity such as closed tickets, lines of code, or PRs are seriously flawed.
- cratermoon 6y ago
  Eventually any simply metric like this will become warped because of the effects of both Goodhart's Law, and Campbell's Law.
  - wsinks 6y ago
    Something that I've been wondering about is that maybe team's should decide what their metrics are for the upcoming 3 months and then come back and decide what the new metrics are. At the very least, it becomes a big game about the org then. Impossible to have 'metric based reviews' at that point. But I think that's fine.
    - nitrogen 6y ago
      That seems to be how quarterly planning worked in the largest org I've seen inside, though the PM and EM did most of the deciding and execs+board approved/disapproved.
  - kovek 6y ago
    How do people learn about a lot of “laws”? I know practically none, and it seems so useful to be able to produce them when explaining an idea...
    - nitrogen 6y ago
      I don't know the answer to your question, but it helps to know what Wikipedia really likes lists, so I was able to guess that it might have an article called "List of eponymous laws", and so it does: https://en.wikipedia.org/wiki/List_of_eponymous_laws https://en.wikipedia.org/wiki/List_of_eponymous_laws
nradov 6y ago
The smallest organizational unit at which productivity can usefully be measured is an agile team of about 7 people. Below that size the effort of quantifying productivity exceeds any possible value of doing so, and incentivizes the wrong behaviors. A good manager can get a reasonable subjective sense of individual productivity but won't be able to quantitatively measure it.
- mundo 6y ago
  I agree, and I would add that there is one good subjective way to measure the productivity of individual developers, and that is talking to their teammates.
deleted 6y ago
[deleted]
tomatohs 6y ago
The problem is that we do not have a standard "output unit." > Productivity: the effectiveness of productive effort, especially in industry, as measured in terms of the rate of output per unit of input. We can all agree "lines of code" is a shit metric, and we can't say "# of bugs closed," because each will have variable difficulty and value. Programmers employed by a business are in charge of automating repetitive tasks, not performing them (the classic measure of productivity). I perform UX research on APIs. Here, we standardize the "output unit" and therefore can get a better idea of a developer's productivity. Every developer performs the same task, so we can simply measure time spent. There will never be an ethical solution to measure developer productivity during the workday; this isn't Ford's assembly line.
- QuercusMax 6y ago
  Even worse, # of bugs closed may be measuring the inverse of what you think you are. See the classic Dilbert cartoon about "writing yourself a new minivan".
  - chrisBob 6y ago
    https://dilbert.com/strip/1995-11-13 https://dilbert.com/strip/1995-11-13 Thank You. I had not seen that one before.
  - convolvatron 6y ago
    I actually had an employee that would write up highly convoluted big reports that I just couldn't understand. he would close them a week later - sometimes with a meaningless patch, sometimes without. couldn't fire him - he was my cofounders toady. the only reason I beat him on 'bugs fixed' is that one of my jobs was to dig through the old issues and remove them if they weren't clear or relevant.
RivieraKid 6y ago
The only way to do it I can think of: have two teams or individuals develop the same thing simultaneously and measure the time required to get a result of the same quality. This should be done in longer term to take into account code quality (poor code quality slows down future development).
- craftinator 6y ago
  A few things can play havoc with this type of measurement. One is that the way we determine the "quality" of the code is based on the current scope of the project. If the scope right now is pull a bunch of values out of spreadsheets and generate reports on them, the highest quality code would be the most terse: it looks up the files, get the information, then displays it. If tomorrow the scope changes to "do that, but in realtime, across multiple machines", the highest quality code is the one that implemented a database and REST API. Since scope changes all the time, we can never evaluate which set of code is the highest quality.
- Lorean 6y ago
  And then keep adding new features (identical for both teams) for a few years and measure the time taken.
- ptr 6y ago
  I did this once for a medium complexity task. The quality ended up the same because both developers had good taste. One developer took 2 hours for the job, the other took 2 weeks. And people don’t believe in 10x developers...
xornox 6y ago
What is productivity of managers? How it is measured? Why productivity of developers must be measured, but productivity of managers not?
- reallydontask 6y ago
  For better or for worse, manager's productivity is normally taken from the developers/engineers the manager manages, namely the team's overall productivity, so we're back to the same problem of how to measure developer's productivity
  - tnli 6y ago
    Yeah, that's messed up. Instead one should take the amount that the manager manages to improve over the baseline. Which might be impossible to measure.
  - alistairSH 6y ago
    My productivity (as a manager) is measured by my ability to deliver on commitments I make with product management and senior leadership. Basically, my ability to match my teams' productivity with corporate goals. Which lines up with nradov's comment about the smallest useful unit for measuring productivity is the typical self-contained development team. Attempting to determine if Bob in TeamA is more productive than Cindy on TeamB doesn't generally result in any actionable information. What matters (from a senior leadership perspective) is can TeamA or TeamB build the things that need to be built in a timeframe that's acceptable to stakeholders. As a manager, if I feel like Bob or Cindy are unproductive, then I need to figure out why. And LoC or number of commits isn't going to tell me that. Possibly the number of defects found in QA, but even that isn't perfect.
learc83 6y ago
We can't even come up with an objective way to score software itself. So how in the world are we going to go even deeper and score the process and the people that create it? Sarah and Bob make clocks, but sometimes they make hats, and sometimes they make screws, or hammers, or lamps. And sometimes the things they make get sold to customers, but sometimes other employees take them home, sometimes they make parts for each other to use when making bigger projects, and often they help each other and other employees out on unrelated projects. And sometimes they do repairs too. Oh yeah they also paint portraits that hang up around the office. Try coming up with a measurement for their individual productivity that is easy enough to be useful, hard to game, and cheap enough to make it worth the price. The first step is to figure out how to measure the value of all the stuff they make...
sytse 6y ago
The article argues that there is no useful measure that operates at a finer grain than “tasks multiplied by complexity”. I think that complexity is hard to measure and therefore easy to game. At GitLab we only measure tasks completed, the number of changes that shipped to production, with the requirement that every change has to add value. This measure has been used throughout R&D https://about.gitlab.com/handbook/engineering/performance-indicators/#rd-overall-mr-rate https://about.gitlab.com/handbook/engineering/performance-in... to assess productivity for multiple years now with good success https://about.gitlab.com/blog/2020/08/27/measuring-engineering-productivity-at-gitlab/ https://about.gitlab.com/blog/2020/08/27/measuring-engineeri... When you tell new engineers about this target they see a great opportunity to game it, just ship smaller changes. It turns out that smaller changes are quicker to ship. Lead to better code and tests. Have lower risk of cancellation and problems in production. And lead to earlier and better feedback. Inspired by Goodhart’s Law I'll propose the following: A measure that when it becomes a target improves productivity. ~Sijbrandij's Law
- jeffbee 6y ago
  It seems like a useful aggregate metric, but is it also used to rate individuals? For that purpose, it seems like it would be terrible. What if you have an experienced staff member from whom everyone else constantly seeks advice? That person may be having a positive impact that isn't visible as merge requests.
  - an_opabinia 6y ago
    If you were a manager, could you honestly say it was ever in a subordinate's best interest to do more than the bare minimum? This should illuminate why these conversations constantly go in circles.
  - sytse 6y ago
    We try to not go below the level of a group when making productivity assessments. So measure the team instead of the individual. This is indeed to encourage helping each-other out as you noted.
choeger 6y ago
You can measure productivity just fine on any tasks that repeat. How long does it take you to run the right tests, find the implementation for a failing test case, make a merge request, create a patch release, pull up the logs in case of an incident? All these tasks repeat over and over again and a good developed can do them much quicker.
WalterBright 6y ago
In every organization I've worked in, it was obvious who the high performers were and who the low performers were. It was obvious to everyone. The only blind spots were people usually seriously misjudged their own performance. The problem, however, is that management is always being pushed to make objective measurements. For example, to fire someone, you have to first put him on an improvement plan with objective measurements. Otherwise, you're wide open to a lawsuit over discrimination, etc. You have to prove to a judge someone isn't performing, or that you gave raises based on performance. Management also gets pushed into these attempts at objective measurements by attempts to optimize the numbers like what works great for a manufacturing process.
awinter-py 6y ago
love the classic 'are story points hours / no / then wtf are they' conversation when PMs intro jira + cousins have never been sure how summing together something that is supposed to have no relationship with time magically provides an estimate of anything also not sure why teams are using the central source of truth for progress as the 'daily todo list making' tool I live in the real world so I estimate in hours
- lgunsch 6y ago
  The answer is in the article itself. It gives you real historical data so you can predict how long the project will take with evidence, rather than just a feeling, hope, or guess. > Velocity is an aggregate measure of tasks completed by a team over time, usually taking into account developers’ own estimates of the relative complexity of each task. It answers questions like, “how much work can this team do in the next two weeks?” The baseline answer is “about as much as they did in the last two weeks,” If there's one thing the last 50 years of software development has conclusively proven, is that estimating the number of man months (hours) a project will take doesn't work.
  - awinter-py 6y ago
    hmm fair, I skimmed and should have read more carefully still: (1) sounds like complexity predicts weeks? so they are estimating hours. And (2) I think if jira clones were really a tool for estimation, they'd have uncertainty scores and some kind of prediction market built in
  - politelemon 6y ago
    A clear solution for this exists: we should double the number of people who are estimating how long a project will take.
  - waynesonfire 6y ago
    counting hours doesn't work so lets count unicorns? Want some of my koolaid? You seem to be out.
- Viliam1234 6y ago
nikolay 6y ago
Most developers like solving problems - this gives them high. Often, without realizing it, they create problems they are eager to solve to get their dose. Solving problems can be quantified, too. Unfortunately, it's hard to quantify the number of problems avoided by manifesting! Often this goes against the first goal I mentioned. For example, solving one problem 10 times gives you closing 10 tickets, making 10 PRs, and contributing a lot more LOCs in a short amount of time. But creating one PR and one ticket, which not only prevents those 10 but 100s and 1,000s more in the future, is quantified as "less work." I've had this at one job recently where every time I suggested fixing a repeated issue was answered with: "We have a bigger fish to fry." Yet, we kept wasting time frying tadpoles.
sharker8 6y ago
Yes if you are running an app development studio where every app looks and functions mostly the same.
siliconc0w 6y ago
I don't get orgs that use stats like commits/LoC/PRs as KPIs. Most time for software engineering ought to spent ensuring you're building the right thing which requires a lot of collaboration, writing design docs, thinking about the problem, etc to avoid 'building the wrong thing' which is probably the 'default' behavior and hard to avoid. Software engineering is only really valuable if you can easily extend and build it on it to enable whatever product or service you're selling to change as the business changes. If you're churning out throw-away code you never reuse you don't realize any of that value and you will lose. I did have the idea of directly tying value to a graph of code that enabled a certain user journey. Sorta like 'CUJ-coverage' instead of test coverage. So if a user spent $20 at checkout, every line of code that was touched to enable that user's journey would be credited with that $20. I think this would be an interesting metric I'd probably respect but there are still probably a lot of blindspots this methodology doesn't capture.
dahart 6y ago
In my ~25 years of professional software development, the single biggest factor in productivity for me has been whether I was involved at the start of a project. Knowing the initial design decisions, and being comfortable changing anything, allows me to be orders of magnitude more productive than when I'm diving into existing code designed by someone else. I saw this perhaps most acutely with a company I sold - for a couple of years I was more productive than on nearly any other large software project I've worked on, because I knew the ins and outs of everything. The developers who bought it and took over are probably better developers than I am, and they are unquestionably excellent coders, yet it took a couple of years for them to get productive at making even medium sized changes. It became incredibly obvious to me how handicapped you are diving into something someone else made, especially if the original designer isn't there anymore. Meshing really well with managers & PMs is probably the next biggest factor in my own experience, but it doesn't come even close to the gap between being there from day 1 vs coming in much later. > Productivity tracking tools and incentive programs will never have as great an impact as a positive culture in the workplace. I'm a fan of choosing to use time management apps and productivity tools to manage my own budgets. But I admit that I hate it when I have to do it for someone else.
- bit_logic 6y ago
  One of the most powerful benefits from being there since the start is the complete confidence in ripping out and deleting obsolete code later. Even good developers new to the project are afraid to do this, and they should be since it's very risky without the full context. The natural trajectory for a project is to keep adding features until it collapses from its own weight. Only the long tenure developer can fight this and revitalize a project by removing the useless excess.
  - qes 6y ago
    I've been with the same company and mostly leading the same software system for the past 10 years. Feature work is such a smaller part of my individual contributions at this point - I do some here and there so I don't get too out of touch with the front end and user experience - but much of my coding work these days is reworking existing core functionality. Thankfully we understand the necessity of deep maintenance for our system that we fully expect to still be running in 10 more years, but even with that it's damned hard to keep up. I can't imagine having developers come and go every couple/few years and little or no leadership support for code and systems improvement.
chadcmulligan 6y ago
This assumes direct managers want productive developers - this is not my experience. The goal of managers is to increase the number of people they manage, and get more money. I have time and again done things fast only to have blocks put in place to slow things down - no one wants the job done easily and go home, where's the money in that. The inability to measure productivity is a direct result of this imho.
knaq 6y ago
Sure, it's easy. Count how many lines of code they write per day. Likewise, aeronautical engineering productivity can be measured by counting kilograms of mass added per day. The real underperformers go negative.
BXLE_1-1-BitIs1 6y ago
I've had a number of projects to either add features or fix a bug in large volumes of truly weird (and sometimes jerkoff code), COLT and JES3 being particularly flagrant examples. It can take weeks to find out where the bad code and less than half a dozen lines to fix the problem. In just about any system of productivity metrics, these two episodes would mark me as dismally productive: In the bank I was working for, the incidence rate of online banking mainframe reIPLs went from every few days to zero. At a telecommunication provider, data center reIPLs similarly reduced.
deleted 6y ago
[deleted]

113 points by 97amarnathk 6y ago

Most software projects get managed with a ticketing system that logs the work to be done as individual tickets. Counting the number of cards a developer closes over a certain period allows us to see what actual work is getting closed off. Measuring closed tickets is an excellent metric if the tasks are written well and assigned based on business priority. When more tickets get closed, more good things are happening with the project, be that bugs getting closed off or features made.

2rsf 6y ago
What you are saying is that my colleague who has been working on one ticket, solving a critical bug, for the last 2 or 3 weeks is a an unproductive one since he haven't closed a ticket for a while ? Counting closed tickets is indeed a measure for something, but by itself it's far from being a good indicator.
- lkxijlewlf 6y ago
  Yes, he's unproductive. What he should have done was broke it up into dozens of tiny pieces so that he could inflate his ticket count. It is more acceptable to the spreadsheet to have "Hard problem part 1", "Hard problem part 2", "Hard problem part ...", than it is to simply have "Hard problem" and take longer to do it.
  - duxup 6y ago
    Hard problem part 1 doesn't really indicate that they did / will solve a problem though. It's work, but is that 'productivity'? That seems kinda arbitrary to just manipulate the issue into tiny pieces to fit some sort of metrics system... but not reflective of the actual work. That seems to just lead to the typical gamification that comes with counting tickets and other metrics systems that end up being arbitrary or even easily manipulated. Ticket measuring just seems like asking for Goodhart’s Law.
    - commandlinefan 6y ago
      In my experience, project managers work as hard as they can to fight that tendency too: they usually insist that tickets be observable and testable independently, specifically to discourage developers from logging time on non-visible tasks.
    - TheCoelacanth 6y ago
      Yeah, you aren't measuring level of productivity. You are measuring level of tolerance for bullshit ticket creation.
  - commandlinefan 6y ago
    I hope you're being sarcastic.
    - TameAntelope 6y ago
      Probably, but to represent the non-sarcastic point of view, it comes down to "[person] in the room" syndrome[1]. Spending 2-3 weeks on a bug is never good, if you're not communicating progress, and one way to communicate progress is to break down the work into smaller chunks. Does it literally have to be individual JIRA tickets? No way, but going off for 2-3 weeks doesn't give the business the insight it needs to in order to wisely invest time/effort into work being executed. [1] https://medium.com/machine-words/a-guy-in-a-room-bbbe058645ef https://medium.com/machine-words/a-guy-in-a-room-bbbe058645e... (I thought Joel Spolsky said this but I can't actually find the original source, if anyone has it I'd appreciate it!)
      - valenterry 6y ago
        > Spending 2-3 weeks on a bug is never good, if you're not communicating progress Assuming that it is important to fix the bug and that the developer is competent and trusted - why not? What would communicating progress improve here? Mind that you cannot communicate when it is done (otherwise it would not be a hard bug) you can only communicate what you have done so far and what you try next. But what kind of business value does that create?
        TameAntelope 6y ago
        The value is that the developer doesn’t reasonably have the context necessary to make the call whether or not, over time, the issue is worth continuing to invest time to resolve. Not that they couldn’t make the call if they had all the info, but the time required to gather and understand all the context would be a second full-time job.
        learc83 6y ago
        That's why the developer has a manager who is a human who can talk to them and find out what they are working on, instead of a robot that can only process tickets created and tickets completed.
  - sokoloff 6y ago
    You can measure progress of a military campaign by counting and reporting on “bullets fired” yet that’s not how military strategists operate.
    - byecomputer 6y ago
      On the other hand, doing busywork is a pretty integral part of being in the military
  - valenterry 6y ago
    Great. Now I can't find anything anymore in the ticket system because every ticket that was non-trivial has been broken down into a dozen of other tickets. There's only one way to call this: ticket system abuse.
- x86amdryzen 6y ago
  Stop using some very rare edge cases as an argument. You are just showing your absolute incompetence.
- matthewmacleod 6y ago
  The mistake there would be getting that specific about it – it's not a useful method for an individual developer, but it _is_ a useful method for the team overall, where these effects get amortised. (Although in this specific case, your colleague who has been working on one ticket for two or three weeks is operating in a way that I find is usually pretty harmful for productivity overall. "Solving a critical bug" is almost universally something that can be broken down further.)
exlurker 6y ago
Nice, I call dibs on the low hanging fruits!
coldcode 6y ago
Paying for dead cobras always pays off. Also ticket != value. Lots of tickets for things that involve almost no work and things that actually make a difference to the customer/product are not equal. Everything I work on is new products/projects and tickets come in all sizes and shapes, and often change daily as some exec crams in more new ideas or some designer or product person "clarifies" the ticket, even after the work is done. Tickets are often written and estimated long before decisions are actually made. Defects are written that require a lot of investigation only to discover it's some other teams problem and you can't do anything or turns out to be a temporary service outage no one communicated or misconfiguration in some CMS or even plain simply not understanding what the product does. Measuring productivity by tickets closed is a whole pile of dead snakes.
- jrs235 6y ago
  In case people don't understand the dead cobras comment: https://en.wikipedia.org/wiki/Cobra_effect https://en.wikipedia.org/wiki/Cobra_effect It pertains to perverse incentives.
- TameAntelope 6y ago
  What do you think of the idea that if tickets aren't doing a good job of representing value to the customer in some form (even if it's second or third order value), those tickets are poorly written and it's a "garbage in/garbage out" situation? It's not really a helpful observation, but I'm curious if there's a way for the relationship between "people asking for things" and "people building things" to be repeatably fruitful (IMO it's very possible that reliably producing customer value is either insanely hard and/or not doable consistently).
  - coldcode 6y ago
    Some things are incredibly important but not directly related to the customer; like services called by service called by services called by clients. It's not always easy to connect the dots especially in a micro service world. But without the ultimate service the customer can't do anything. Making good tickets across a dozen organizations and 100's of people is hard to ever get right. Which is why counting tickets is sort of pointless, you might have a ticket to add a single value to a database and without it, the whole product doesn't work, but you have no idea since there are 10 layers between you and the real customer.
zarkov99 6y ago
I think there is some value to that metric but I would not call it excellent. A really good developer might come up with a slight requirement change or an engineering detour that makes many tickets meaningless, he might improve tooling such that things that took hours now take minutes, he might be able to write tests or come up with new processes that increase quality 10x. If we think of guys like Jeff Dean, the ability to close tickets written by project managers is not what makes them stand out.
- commandlinefan 6y ago
  Well, don't forget you're supposed to "estimate" everything (based on a 10-second glance at the description) before you do it, and you're also measured on the accuracy of your estimates.
danjac 6y ago
That assumes all tickets are the same. A developer might take on a very difficult task with a lot of hidden technical complexity that ties them up for weeks. Another might pick up little bugs and small text updates. With no other insight, the metric is meaningless. It becomes easy to game by avoiding any difficult and time consuming tickets - such as refactoring - as much as possible, instead picking the quick and easy things that make your metric look good.
908B64B197 6y ago
> Measuring closed tickets is an excellent metric if the tasks are written well and assigned based on business priority Don't forget the weight. I've seen single tickets taking weeks for bug investigation.
harryf 6y ago
Sorry but I strongly disagree. In fact I’ll come out and say its perhaps one of the worst ways you can measure productivity. At best you’re measuring _activity_ not productivity. You just turned a group of smart people into headless chickens jumping on whatever ticket so they can to look busy. Which cultivates an environment of fear, which in turn kills deep thought and creativity... two essential ingredients for good software. I could even argue that ticketing systems are the bane of good software, making real priorities intransparent... but that’s a rabbit hole I won’t go into here. Instead I’d argue we shouldn’t be trying to measure developer productivity at all. Productivity in software development is non-linear and difficult to assign individually. How do you measure the productivity of that “lazy guy” that had an amazing shower thought one morning, implemented it by lunchtime, which in turn leads to the company making millions more by the end of the year? Or what about the person on the team that spends most of their time supporting the rest of the team, unblocking them and helping them be productive? Two examples of why we shouldn’t even be trying to measure developer productivity. My own experience after 25 years in this industry is the moment someone says “but how do we measure developer productivity?” is the moment that companies software products begins a long, slow death. Ultimately what development teams and companies (not individuals) should be measured on is _results_ that positively impact customers and business. When the product is a success, no one cares about individual productivity.

BrandonMarc 6y ago

He mentions increasing salary won't lead to increased productivity ... and that's true, if the same developer remains. But what if we remove that constraint? What if increased salary means a higher quality of developer takes the position? Wouldn't this mean higher productivity? Bit of a cold scenario, but one way to game it out is hypothetically removing the current dev and then hiring someone better at double the pay. Or, less unfair seeming, double the pay by hiring a second dev. That might not double productivity ... depending on the situation it might 1.5x it, or just as easily 4x it.

digitalsushi 6y ago
Two engineers who are not aligned with each other's goals will easily do the work of zero people.
- gootdude 6y ago
  Well said, this point can not be stated enough!
kemiller2002 6y ago
I don't completely agree with this. Money can affect developer productivity, but it takes time to realize its value. Money makes things easier. That person whose being paid the smallest amount possible, is probably trying to just scrape by. Paying people more money frees their personal life up to handle other things, less stress about family stuff etc. I never believed the "leave your personal life at the door," thing. How you live affects your work, and making life easier makes people more productive (on average, not all cases). Now this isn't an immediate thing. It's an investment and it takes time. The person has to realize that money is available (emotionally) and start to trust it will be there.
suprfnk 6y ago
> a higher quality of developer So what is a "higher quality of developer"?
throwaway0950 6y ago

qz2 6y ago

Developer productivity is inversely proportional to complaining. Listen to complaints carefully.

digitalsushi 6y ago
I saw your text as a light grey so I decided to re-read it a few times. I absolutely agree with you. The people who complain the least should be paid the most attention when they do.
- kemiller2002 6y ago
  There is a lecture by the late Randy Pausch (https://www.youtube.com/watch?v=ji5_MqicxSo&vl=en https://www.youtube.com/watch?v=ji5_MqicxSo&vl=en). The gist of the part I'm mentioning is, "When I stop correcting you, I've given up on you." People who don't voice their opinions aren't necessarily happy, they quite possibly have decided it's not worth trying to change things.
  - lmilcin 6y ago
    Yes, I think promoting people who DO NOT complain is problematic. There is no right way to deal with this other than to listen to complains and figure out what it is. One way of thinking about complaining is that it is a form of feedback. As a manager, you don't want to silence people giving you feedback, frankly, this is about as stupid thing as you can do. Better way to deal with complaints is to educate on what kind of complaints are productive and what kinds are destructive. For example, I try (not always succeed) to restrict to myself to only complain about things that I am ready to solve if somebody tells me "go ahead, fix it".
  - Viliam1234 6y ago
    Funny how this can reinforce the idea that only incompetent people complain. Imagine a situation where the company and/or the project have a few serious problems, but the company refuses to fix or even admit any of that. The developers who couldn't live with the problems have already quit. The developers who remained have stopped complaining, because they have given up. A new developer comes, notices the problems, and starts complaining about them. People notice that the newbie makes a fuss, but nothing changes. Later the developer either quits, or gets used to it and stops complaining. Here is how the management probably interprets the situation: "People with the least experience complain most. The correct approach is to ignore them, and wait for them to grow up. More experienced developers have realistic expectations and mature behavior."

burade 6y ago

Any decently competent technical leader can tell if a developer is being productive or not. It's stupid to waste time trying to measure something that is virtually unmeasurable.

NortySpock 6y ago
"It's unmeasurable, but everyone can tell." is that what you're saying? Seems like a No True Scotsman fallacy to say only good technical leaders can tell if a developer is being productive and in the same breath say it's unmeasurable. hours worked, bugs fixed, tickets closed, costs saved, clients saved, KPIs/OKRs hit, time in queue, hours-to-close-ticket, uptime, SLAs hit... surely some collection of indicators, while not a pure signal, would let you highlight outliers either above or below the curve.
- valenterry 6y ago
  The funny thing is, except for "tickets closed" and maybe "costs saved" there is no metric here you can directly attribute to a single developer, and probably not even a team. Even "hours worked". What do you mean, hours spent in the office? How do you know the person was not drinking coffee or staring at their code in thought of something else. That being said, I think your metrics are good. And even single developers should be measured by them (which means all developers get measured by the same metric and get the same value). Why? Because it helps the business of SLAs are hit, no matter why they are hit.
- astrobe_ 6y ago
  All of your indicators mean absolutely nothing if you do not measure the difficulty of the tasks submitted. Do that, then we can talk. It's like saying, "your productivity as a carpenter is how many houses you build in a year" while refusing to take into account how big or complex to build are those houses.
- deleted 6y ago
  [deleted]
rajacombinator 6y ago

jhunter1016 6y ago

Productivity is important, sure. But as with all other professions in which people interact, the interpersonal skills and behavior tend to be more important IMO. Productivity can be massively impacted (positively and negatively) by how well people communicate and get along with each other.

forbushbl 6y ago

As an individual I often wonder if my contributions are meaningful. The author says, “individual performance is best left for individual contributors to measure in themselves and each other.” How can individuals possibly measure their own performance if it can’t be measured externally?

valenterry 6y ago
Of course it can be measured externally, it is just too expensive.

anxiostial 6y ago

It is pretty ironic how much time is wasted on trying measure developer productivity.

orky56 6y ago
Why is it ironic? The stakeholders for increased developer productivity go beyond just developers. Even the slightest increase in developer productivity, let alone the ability to objectively measure it, is the holy grail of software development. Companies with access to nearly infinite resources can and would deploy them for a marginal gain in developer productivity. So much emphasis is spent on hiring the most brilliant minds and then on managing their projects and time so why not on optimizing their output.
- anxiostial 6y ago
  You are making a lot of statements here without anything to really back it up, why is it the holy grail of software development? measuring output and actually improving developer performance are not even remotely the same thing, would it not make more sense to spend time on anything related to the actual development, like training developers for example?

ExcavateGrandMa 6y ago

measuring the copy/paste productivity is bad for the mind :Ð KrkkrkrkrKRKrkrkrkr.

GartzenDeHaes 6y ago

One of the most useful programmer metrics that I've found is code churn: (new lines + deleted lines) / total changed lines. Instead of telling you how much work your programmers are doing, this metric tells you what kind of work your programmers are doing. Small numbers mean bug fixing (end of project and maintenance) and large numbers mean new development and features.

sixstringtheory 6y ago
What about high deletion amounts? I've merged PRs with hundreds of thousands of lines deleted and none added. It took quite a bit of sleuthing to figure out someone had left entire copies of directories side by side with different names, where one was completely unused. Conversely, someone had a huge addition to the repo that actually was total garbage.
- GartzenDeHaes 6y ago
  A large change to the code base would indicate to me that the product isn't ready to ship or test, possibility. At the very least, it would prompt a discussion about the code and how we are managing it. Maybe we need some process changes or tools to prevent junk code from accumulating. Edit: if you're talking about the math, I think "changed" includes added and deleted. So, it's the ratio of added and deleted to the total change.

rileymat2 6y ago

I have never felt my individual productivity go up. It feels like as I progress my individual work stays the same, but helping others eats any efficiency gains I personally make. As if when you are new to a module, you are slow because you don’t know anything, then once you have expertise, you are slow because you know everything and are helping others. Would be interesting to measure this somehow.

orky56 6y ago
This is the only way you can scale your time, by ramping up others to be as efficient as you are. Although you might be becoming less of a contributor individually, you are enabling the larger group. This type of productivity can definitely be tracked based on how many people you have helped and their corresponding lineage of knowledge and work output.
Viliam1234 6y ago
> As if when you are new to a module, you are slow because you don’t know anything, then once you have expertise, you are slow because you know everything and are helping others. This suggests that the proper way to keep team productivity high is to have all team members working on the product since the beginning, and treat them well so that they don't quit and don't have to be replaced by new ones. Maybe even start with slightly more people on the project than necessary, so if a few of them quit during the project for unrelated reasons, you can still finish the project with the remaining ones. Probably not going to happen, because this goes against maximizing short-term productivity at the beginning of the project. The short-term productivity is maximized by having the team as small as possible, and only worrying about problems after they happen.

carapace 6y ago

I once talked to a retired hardware engineer, a fellow who made real electronic devices, not software. He told me that, over the whole course of his career, 80% of the projects he worked on never made it to market. In other words, 4/5th of his total "productivity" turned out to be waste. Make of it what you will.

BrandonMarc 6y ago
Put differently, 80% of his time gave his brain training and practice which likely improved the 20% that made it to market. Also consider, frustrating though I'm sure that was, he probably still got paid for his effort in the 80%.
- edelans 6y ago
  Interesting point of view, thanks for putting it that way ! It makes me think of artists : - how many hours did a musician spent on his instrument before selling his first record ? - How many drawings/paintings before Picasso could sell something ? (etc.)
1123581321 6y ago
He could not have chosen to only attempt the successful projects.
- deleted 6y ago
  [deleted]
triceratops 6y ago
Why didn't he just work on projects that would make it to market? /s

HourglassFR 6y ago

Maybe when developer productivity measurment becomes standard accross the industry we will realise that tech workers are in fact workers. Cogs in a machine. And not independant individuals imposing their will to the world through sheer will like some Randian hero. Maybe then it will then be plainly evident that developers are as alienated as any service worker, and in the end as disposable in the eyes of the shareholders. Will we then organize with other workers to create better working conditions for everyone or will there be fewer and fewer developers working with ever more powerful technology chasing richer than ever VCs?

908B64B197 6y ago
> as disposable in the eyes of the shareholders. If a company is willing to sacrifice engineering talent and institutional knowledge for short term gains... Good luck staying in business. Reference: Every outsourcing project I've seen.
- rusticpenn 6y ago
  They are called Best cost countries now.
yourapostasy 6y ago
> Maybe when developer productivity measurment becomes standard accross the industry... I suggest you look to database models/schemas standardization for an indication of how close this is coming to fruition. I personally can't measure developer productivity at a fine-grained level until requirements are stabilized, and I personally cannot stabilize requirements unless the domain is so well known the data store is standardized. I had hoped SAP would lead the charge through empirically iterating towards standards, but they left out the huge small and mid-size business markets with what they use today. And what they use today is still far from industries' standards. We're no closer to standardization than when I started in software decades ago. We don't even have standard means of storing, transforming, displaying and tracking metadata upon calendars, addresses, phone numbers, names, and lots of other ephemera I can rattle off, within a single stakeholder industry, not to speak of within the software industry in general. There have certainly been efforts to standardize like Silverston's, but they haven't caught traction. I'd sure like to see that happen, because it would short-circuit a lot of the discussions I engage with stakeholders to only the site-specific requirements, where I really add business value. Instead, I have to derive the data model from intricate discussion of their requirements, since they themselves have not agreed upon the parts that are common across their respective industries, so I end up at the start of dicussions with all sorts of little twisty pieces of a data model, all alike.

jeffbee 6y ago

Productivity as a software developer consists chiefly in not making mistakes. That can lead to the situation where your best developers may appear to do nothing for long stretches. Research and deliberation are desirable. Blind hacking is the least valuable yet most visible activity of inexperienced programmers. All common "objective" measures of productivity such as closed tickets, lines of code, or PRs are seriously flawed.

cratermoon 6y ago
Eventually any simply metric like this will become warped because of the effects of both Goodhart's Law, and Campbell's Law.
- wsinks 6y ago
  Something that I've been wondering about is that maybe team's should decide what their metrics are for the upcoming 3 months and then come back and decide what the new metrics are. At the very least, it becomes a big game about the org then. Impossible to have 'metric based reviews' at that point. But I think that's fine.
  - nitrogen 6y ago
    That seems to be how quarterly planning worked in the largest org I've seen inside, though the PM and EM did most of the deciding and execs+board approved/disapproved.
- kovek 6y ago
  How do people learn about a lot of “laws”? I know practically none, and it seems so useful to be able to produce them when explaining an idea...
  - nitrogen 6y ago
    I don't know the answer to your question, but it helps to know what Wikipedia really likes lists, so I was able to guess that it might have an article called "List of eponymous laws", and so it does: https://en.wikipedia.org/wiki/List_of_eponymous_laws https://en.wikipedia.org/wiki/List_of_eponymous_laws

nradov 6y ago

The smallest organizational unit at which productivity can usefully be measured is an agile team of about 7 people. Below that size the effort of quantifying productivity exceeds any possible value of doing so, and incentivizes the wrong behaviors. A good manager can get a reasonable subjective sense of individual productivity but won't be able to quantitatively measure it.

mundo 6y ago
I agree, and I would add that there is one good subjective way to measure the productivity of individual developers, and that is talking to their teammates.

deleted 6y ago

[deleted]

tomatohs 6y ago

The problem is that we do not have a standard "output unit." > Productivity: the effectiveness of productive effort, especially in industry, as measured in terms of the rate of output per unit of input. We can all agree "lines of code" is a shit metric, and we can't say "# of bugs closed," because each will have variable difficulty and value. Programmers employed by a business are in charge of automating repetitive tasks, not performing them (the classic measure of productivity). I perform UX research on APIs. Here, we standardize the "output unit" and therefore can get a better idea of a developer's productivity. Every developer performs the same task, so we can simply measure time spent. There will never be an ethical solution to measure developer productivity during the workday; this isn't Ford's assembly line.

QuercusMax 6y ago
Even worse, # of bugs closed may be measuring the inverse of what you think you are. See the classic Dilbert cartoon about "writing yourself a new minivan".
- chrisBob 6y ago
  https://dilbert.com/strip/1995-11-13 https://dilbert.com/strip/1995-11-13 Thank You. I had not seen that one before.
- convolvatron 6y ago
  I actually had an employee that would write up highly convoluted big reports that I just couldn't understand. he would close them a week later - sometimes with a meaningless patch, sometimes without. couldn't fire him - he was my cofounders toady. the only reason I beat him on 'bugs fixed' is that one of my jobs was to dig through the old issues and remove them if they weren't clear or relevant.

RivieraKid 6y ago

The only way to do it I can think of: have two teams or individuals develop the same thing simultaneously and measure the time required to get a result of the same quality. This should be done in longer term to take into account code quality (poor code quality slows down future development).

craftinator 6y ago
A few things can play havoc with this type of measurement. One is that the way we determine the "quality" of the code is based on the current scope of the project. If the scope right now is pull a bunch of values out of spreadsheets and generate reports on them, the highest quality code would be the most terse: it looks up the files, get the information, then displays it. If tomorrow the scope changes to "do that, but in realtime, across multiple machines", the highest quality code is the one that implemented a database and REST API. Since scope changes all the time, we can never evaluate which set of code is the highest quality.
Lorean 6y ago
And then keep adding new features (identical for both teams) for a few years and measure the time taken.
ptr 6y ago
I did this once for a medium complexity task. The quality ended up the same because both developers had good taste. One developer took 2 hours for the job, the other took 2 weeks. And people don’t believe in 10x developers...

xornox 6y ago

What is productivity of managers? How it is measured? Why productivity of developers must be measured, but productivity of managers not?

reallydontask 6y ago
For better or for worse, manager's productivity is normally taken from the developers/engineers the manager manages, namely the team's overall productivity, so we're back to the same problem of how to measure developer's productivity
- tnli 6y ago
  Yeah, that's messed up. Instead one should take the amount that the manager manages to improve over the baseline. Which might be impossible to measure.
- alistairSH 6y ago
  My productivity (as a manager) is measured by my ability to deliver on commitments I make with product management and senior leadership. Basically, my ability to match my teams' productivity with corporate goals. Which lines up with nradov's comment about the smallest useful unit for measuring productivity is the typical self-contained development team. Attempting to determine if Bob in TeamA is more productive than Cindy on TeamB doesn't generally result in any actionable information. What matters (from a senior leadership perspective) is can TeamA or TeamB build the things that need to be built in a timeframe that's acceptable to stakeholders. As a manager, if I feel like Bob or Cindy are unproductive, then I need to figure out why. And LoC or number of commits isn't going to tell me that. Possibly the number of defects found in QA, but even that isn't perfect.

learc83 6y ago

We can't even come up with an objective way to score software itself. So how in the world are we going to go even deeper and score the process and the people that create it? Sarah and Bob make clocks, but sometimes they make hats, and sometimes they make screws, or hammers, or lamps. And sometimes the things they make get sold to customers, but sometimes other employees take them home, sometimes they make parts for each other to use when making bigger projects, and often they help each other and other employees out on unrelated projects. And sometimes they do repairs too. Oh yeah they also paint portraits that hang up around the office. Try coming up with a measurement for their individual productivity that is easy enough to be useful, hard to game, and cheap enough to make it worth the price. The first step is to figure out how to measure the value of all the stuff they make...

sytse 6y ago

The article argues that there is no useful measure that operates at a finer grain than “tasks multiplied by complexity”. I think that complexity is hard to measure and therefore easy to game. At GitLab we only measure tasks completed, the number of changes that shipped to production, with the requirement that every change has to add value. This measure has been used throughout R&D https://about.gitlab.com/handbook/engineering/performance-indicators/#rd-overall-mr-rate https://about.gitlab.com/handbook/engineering/performance-in... to assess productivity for multiple years now with good success https://about.gitlab.com/blog/2020/08/27/measuring-engineering-productivity-at-gitlab/ https://about.gitlab.com/blog/2020/08/27/measuring-engineeri... When you tell new engineers about this target they see a great opportunity to game it, just ship smaller changes. It turns out that smaller changes are quicker to ship. Lead to better code and tests. Have lower risk of cancellation and problems in production. And lead to earlier and better feedback. Inspired by Goodhart’s Law I'll propose the following: A measure that when it becomes a target improves productivity. ~Sijbrandij's Law

jeffbee 6y ago
It seems like a useful aggregate metric, but is it also used to rate individuals? For that purpose, it seems like it would be terrible. What if you have an experienced staff member from whom everyone else constantly seeks advice? That person may be having a positive impact that isn't visible as merge requests.
- an_opabinia 6y ago
  If you were a manager, could you honestly say it was ever in a subordinate's best interest to do more than the bare minimum? This should illuminate why these conversations constantly go in circles.
- sytse 6y ago
  We try to not go below the level of a group when making productivity assessments. So measure the team instead of the individual. This is indeed to encourage helping each-other out as you noted.

choeger 6y ago

You can measure productivity just fine on any tasks that repeat. How long does it take you to run the right tests, find the implementation for a failing test case, make a merge request, create a patch release, pull up the logs in case of an incident? All these tasks repeat over and over again and a good developed can do them much quicker.

WalterBright 6y ago

In every organization I've worked in, it was obvious who the high performers were and who the low performers were. It was obvious to everyone. The only blind spots were people usually seriously misjudged their own performance. The problem, however, is that management is always being pushed to make objective measurements. For example, to fire someone, you have to first put him on an improvement plan with objective measurements. Otherwise, you're wide open to a lawsuit over discrimination, etc. You have to prove to a judge someone isn't performing, or that you gave raises based on performance. Management also gets pushed into these attempts at objective measurements by attempts to optimize the numbers like what works great for a manufacturing process.

awinter-py 6y ago

love the classic 'are story points hours / no / then wtf are they' conversation when PMs intro jira + cousins have never been sure how summing together something that is supposed to have no relationship with time magically provides an estimate of anything also not sure why teams are using the central source of truth for progress as the 'daily todo list making' tool I live in the real world so I estimate in hours

lgunsch 6y ago
The answer is in the article itself. It gives you real historical data so you can predict how long the project will take with evidence, rather than just a feeling, hope, or guess. > Velocity is an aggregate measure of tasks completed by a team over time, usually taking into account developers’ own estimates of the relative complexity of each task. It answers questions like, “how much work can this team do in the next two weeks?” The baseline answer is “about as much as they did in the last two weeks,” If there's one thing the last 50 years of software development has conclusively proven, is that estimating the number of man months (hours) a project will take doesn't work.
- awinter-py 6y ago
  hmm fair, I skimmed and should have read more carefully still: (1) sounds like complexity predicts weeks? so they are estimating hours. And (2) I think if jira clones were really a tool for estimation, they'd have uncertainty scores and some kind of prediction market built in
- politelemon 6y ago
  A clear solution for this exists: we should double the number of people who are estimating how long a project will take.
- waynesonfire 6y ago
  counting hours doesn't work so lets count unicorns? Want some of my koolaid? You seem to be out.
Viliam1234 6y ago

nikolay 6y ago

Most developers like solving problems - this gives them high. Often, without realizing it, they create problems they are eager to solve to get their dose. Solving problems can be quantified, too. Unfortunately, it's hard to quantify the number of problems avoided by manifesting! Often this goes against the first goal I mentioned. For example, solving one problem 10 times gives you closing 10 tickets, making 10 PRs, and contributing a lot more LOCs in a short amount of time. But creating one PR and one ticket, which not only prevents those 10 but 100s and 1,000s more in the future, is quantified as "less work." I've had this at one job recently where every time I suggested fixing a repeated issue was answered with: "We have a bigger fish to fry." Yet, we kept wasting time frying tadpoles.

sharker8 6y ago

Yes if you are running an app development studio where every app looks and functions mostly the same.

siliconc0w 6y ago

I don't get orgs that use stats like commits/LoC/PRs as KPIs. Most time for software engineering ought to spent ensuring you're building the right thing which requires a lot of collaboration, writing design docs, thinking about the problem, etc to avoid 'building the wrong thing' which is probably the 'default' behavior and hard to avoid. Software engineering is only really valuable if you can easily extend and build it on it to enable whatever product or service you're selling to change as the business changes. If you're churning out throw-away code you never reuse you don't realize any of that value and you will lose. I did have the idea of directly tying value to a graph of code that enabled a certain user journey. Sorta like 'CUJ-coverage' instead of test coverage. So if a user spent $20 at checkout, every line of code that was touched to enable that user's journey would be credited with that $20. I think this would be an interesting metric I'd probably respect but there are still probably a lot of blindspots this methodology doesn't capture.

dahart 6y ago

In my ~25 years of professional software development, the single biggest factor in productivity for me has been whether I was involved at the start of a project. Knowing the initial design decisions, and being comfortable changing anything, allows me to be orders of magnitude more productive than when I'm diving into existing code designed by someone else. I saw this perhaps most acutely with a company I sold - for a couple of years I was more productive than on nearly any other large software project I've worked on, because I knew the ins and outs of everything. The developers who bought it and took over are probably better developers than I am, and they are unquestionably excellent coders, yet it took a couple of years for them to get productive at making even medium sized changes. It became incredibly obvious to me how handicapped you are diving into something someone else made, especially if the original designer isn't there anymore. Meshing really well with managers & PMs is probably the next biggest factor in my own experience, but it doesn't come even close to the gap between being there from day 1 vs coming in much later. > Productivity tracking tools and incentive programs will never have as great an impact as a positive culture in the workplace. I'm a fan of choosing to use time management apps and productivity tools to manage my own budgets. But I admit that I hate it when I have to do it for someone else.

bit_logic 6y ago
One of the most powerful benefits from being there since the start is the complete confidence in ripping out and deleting obsolete code later. Even good developers new to the project are afraid to do this, and they should be since it's very risky without the full context. The natural trajectory for a project is to keep adding features until it collapses from its own weight. Only the long tenure developer can fight this and revitalize a project by removing the useless excess.
- qes 6y ago
  I've been with the same company and mostly leading the same software system for the past 10 years. Feature work is such a smaller part of my individual contributions at this point - I do some here and there so I don't get too out of touch with the front end and user experience - but much of my coding work these days is reworking existing core functionality. Thankfully we understand the necessity of deep maintenance for our system that we fully expect to still be running in 10 more years, but even with that it's damned hard to keep up. I can't imagine having developers come and go every couple/few years and little or no leadership support for code and systems improvement.

chadcmulligan 6y ago

This assumes direct managers want productive developers - this is not my experience. The goal of managers is to increase the number of people they manage, and get more money. I have time and again done things fast only to have blocks put in place to slow things down - no one wants the job done easily and go home, where's the money in that. The inability to measure productivity is a direct result of this imho.

knaq 6y ago

Sure, it's easy. Count how many lines of code they write per day. Likewise, aeronautical engineering productivity can be measured by counting kilograms of mass added per day. The real underperformers go negative.

BXLE_1-1-BitIs1 6y ago

I've had a number of projects to either add features or fix a bug in large volumes of truly weird (and sometimes jerkoff code), COLT and JES3 being particularly flagrant examples. It can take weeks to find out where the bad code and less than half a dozen lines to fix the problem. In just about any system of productivity metrics, these two episodes would mark me as dismally productive: In the bank I was working for, the incidence rate of online banking mainframe reIPLs went from every few days to zero. At a telecommunication provider, data center reIPLs similarly reduced.

deleted 6y ago

[deleted]