12 ms·

A useful productivity measure?

249 points by rglullis 2y ago

vouaobrasil 2y ago
> In other words, in the absence of RoI measures, the percent of engineering time spent on value-add activities is a pretty good proxy for productivity. What about using self-rated satisfaction of your workers in terms of how happy they are with their engineering solutions? I find that when I solve a really nice problem and add something that is cool, or fix a very annoying bug, I feel very satisfied with the result. Whereas, I have produced things in the past (when I worked as a programmer) that seemed to make management happy that I wasn't satisfied with, and it seemed to me at least that the results weren't as useful in the end.
- yen223 2y ago
  To put it bluntly, Goodhart's law will really kick your ass if you went with that.
  - deleted 2y ago
    [deleted]
- mjr00 2y ago
  This is way too subjective; I've met way too many developers throughout my career who loved to spend their time crafting "perfect" code rather than deliver customer-facing value for this to work as a measure that aligns with business objectives.
- jimbokun 2y ago
  Useful to know, but different from productivity.
Archelaos 2y ago
> * The value it’s estimated to generate > * The amount we’re willing to bet A reasonable bet should correspond to the (estimated) probability of its outcome. So the amount someone is willing to bet should directly reflect the value it is estimated to generate. Betting higher or lower makes no sense.
- datadrivenangel 2y ago
  It depends on your risk tolerance! We may want to bet more for more likely outcomes, even if the expected return is lower, if we value predictability and the cost of a bad portfolio outcome is high.
lifeisstillgood 2y ago
I think there are two answers here. The first is treating software develolment as a process - and using some form of statistical process control to “manage” the process - reduce waste etc. this is it turns out exactly what Agile / Scrum was / is all about. Remember those “retrospectives” we are supposed to do - that’s the point at which we whip out a diagram and see if we are trending out of band. It’s fine. It works really and it does help reduce waste - which for any sane world reduced waste is a proxy for productivity. But no matter what that is only “doing things right”. It is not “doing the right thing”. At this point we can also say we have tactical and operational parts under control (or at least monitored). The “doing the right thing” (or strategy) is next and this means something most business refuse to accept - that their grand strategy, their Vision, might be total bollocks. Searching for productivity in software teams is the new version of blaming lazy workers in the factory floor - we are all software ompanies now, and if you are not succeeding you can either blame your crappy product market fit, your strategy or you can whip the employees harder. Guess which is easier. So if you and the CEO need to make fucking appointments to have a honest chat about strategy something has gone terribly wrong with your companies ability to quickly respond strategically - if you have a problem, it’s not the employees that need to run faster. edit: yeah I started ranting there, sorry. The whole “takes ages to get face to face time” is a red flag, but you can get around it. As for the rest, I really respect the attempt to treat the whole company as a system - not just the software team - the product bet idea clearly sets out goals and places emphasis on whole org. That’s really great but as you say “executives prefer if other people are held accountable”. But if they can learn a company can be programmable (then they might learn to start programming)
- robertlagrant 2y ago
  > this means something most business refuse to accept - that their grand strategy, their Vision, might be total bollocks Actually this is often talked about, but only internally to the senior team. You generally don't want to be saying "actually widgets might be a bad idea" which might horribly demoralise your widget factory workers, when you might well conclude after discussion and analysis that actually they're still a good idea.
didgeoridoo 2y ago
The “fatal flaw” of skipping maintenance to max out value-add work seems like it could be addressed by ensuring that any accumulated tech debt be properly accounted for, rather than swept under the carpet. You’d consider “productivity” then to be value-add work minus identified tech debt. Since calling out tech debt would hurt leadership’s own metrics, you’d need a mechanism to allow individual engineers to blamelessly (and probably anonymously) report, validate, and attach a cost metric to this tech debt. The org would then be incentivized to balance work that delivers immediate value with work that represents an investment in making the team more efficient and effective at delivering that value in the long run.
- datadrivenangel 2y ago
  That's the thing though! Quality :tm: is very hard to account for, especially when the quality is in the system, tooling, and process to build the quality end product. And often times technical debt isn't actually something you can put on the balance sheet or bug tracker. It's all the little investments in the future that are deferred or skipped. That one code change is so minor you can phone in the review, you'll write better documentation for that new feature in a few weeks when you have some time, etc.
  - didgeoridoo 2y ago
    It’s a good practice to note things for later and why they should be done though, right? Even if it is never intended to be worked on, noting that you would write more docs for this class but you don’t have time is an important indicator of productive capacity for leadership. If I start seeing a lot of that as an executive, I should start to worry if we’re building our value-add on a foundation of sand.
  - duderific 2y ago
    > you'll write better documentation for that new feature in a few weeks when you have some time The trick here is to make a ticket for it. At least then, it can be prioritized appropriately, instead of disappearing into a void.
roenxi 2y ago
I really like this article so I am sad to be cynical about it. Nevertheless. If you're engaging in C-suite politics as described in this article, what on earth would be the motivation for writing all this in public? I doubt the situation will evolve favourably to Mr Shore if the leadership team starts reading this. Full agreement with the perspective though. CEOs (& consultant friends) generally have some extremely reasonable expectations about software engineer productivity based on experiences in manufacturing, primary industries, healthcare, service and public sectors. Surely we need accountability and productivity metrics! It is a long and painful process that may never end watching them gathering the evidence needed to learn that software engineer productivity is different. A sad history repeating over and over again like clockwork. This CTO seems to be finding a golden path for how to manage expectations without beating that over people's heads and I'm taking notes. I don't think it'll work out exactly the way he hopes but I love the attempt at making his team productive.
- simonw 2y ago
  If I was a CEO with a VP who wrote something this realistic and honest, my response would be to pressure the rest of my execs to give me that same level of quality information. Breath of fresh air!
- AtlasBarfed 2y ago
  One of my long standing beefs with C-suites and their MBAs and whatever meat factory grinds them out is the continued lack of comprehension of basic software and IT management. Back in the early 2000s, this could be saliently characterized by the exuberance many CEOs of companies bragged about how IT-ignorant they were. I saw the "my secretary prints out my emails" written proudly in print many a time in the CEO worship magazines like Fortune and the like. Every decade of continued IT penetration and performance benefits in business are continuing the principle: the health of your business is strongly tied to its IT health. BUT MY COMPANIES JUST STAMPS WIDGETS. Ok, look. MBAs long ago accepted that every business, regardless of what it does, needs two things: accounting and finance. So every MBA gets a decent education on those. Well, hate to tell you, MBAs need to know the fundamentals of IT. Basically, C-suites not understanding productivity measures in IT are hard is stupid not only because it is a basic aspect of IT management, but its a bit more disturbing than that: YOU CAN'T MEASURE MANAGER PRODUCTIVITY EITHER. How does traditional MBA measure middle managers and those types of orgs? That is rife with all the problems of measuring developers, not the least of it being the "juke the stats" universal phenomenon. That's why the most important single indicator of manager ability is: headcount. Headcount headcount headcount. The second is size of budget: bigger bigger bigger. Of course everyone that has working in companies knows the hilarity of what that produces: bloated orgs, pointless spending to use up a budget otherwise it gets cut, building empires, etc. The essence of developer that C-suites need to understand is that they are not factory workers: they are in fact MANAGERs. They just don't manage people, they manage virtual employees/subordinates (software). Go ahead, ask any software dev in an org what they do. Generally, somewhere in there is a bunch of acronyms and the fact that they ... wait for it ... MANAGE those acronyms. Krazam's microservices is a perfect example of this. What is that long diatribe essentially about? MANAGING all those services. That gets to another major undercurrent of IT. Managers want to place IT employees in the "worker bee" category. They don't want IT workers to be managers, and entitled to the great benefits according the management class. The long running IT salary advantage in the US, and the constant ebb and flow between management and IT "working" to push that down is a major social undercurrent here.
thcipriani 2y ago
Seems like the author of this is aware of Goodhart's law, but it bears repeating: both of the proposed real measures (RoI and value-add capacity) could come at the expense of uptime and stability. John Doerr talks about balancing OKRs, so you could have you product bets and...something about stability: hitting SLAs, whatever. And the term "product bets" makes the leadership team sound like Mortimer and Randolph Duke.
haskellandchill 2y ago
What about the Bayesian methods shown in "How to Measure Anything"? They have been applied to Cybersecurity ("How to Measure Anything in Cybersecurity Risk" in a very thorough and convincing manner. It looks like the business around it is trying to apply it to product management (https://hubbardresearch.com/shop/measure-anything-project-management-seminar-plus/ https://hubbardresearch.com/shop/measure-anything-project-ma...). Basically the idea is when things are hard to measure we should not abandon quantitative scales and use qualitative ones (like t-shirt sizes) but instead use probabilities to quantify our uncertainty and leverage techniques like bayesian updates, confidence intervals, and monte carlo simulations.
- apwheele 2y ago
  This is not inconsistent with How to Measure Anything IMO (I like that book as well). The biggest issue to me is that he does not define actual follow ups on ROI -- it is all estimated in this framework. So it is all good to define how to prioritize, it is not helpful though retrospectively to see if people are making good estimates. My work very rarely is a nicely isolated new thing -- I am building something on-top of an already existing product. In these scenarios ROI is more difficult -- you need some sort of counterfactual profit absent the upgrade. Most people just take total profit for some line, which is very misleading in the case of incremental improvements.
  - haskellandchill 2y ago
    The problem is the muda should have expected values associated with them. Bugs and security vulnerabilities do cost money, these are 90% confidence intervals of dollar impact from How to Measure.
- dentemple 2y ago
  Do you have any links to back up your information that doesn't cost $17,250 to view?
  - haskellandchill 2y ago
ltbarcly3 2y ago
[flagged]
- senorrib 2y ago
  This comment is so incredibly useless that it’s hard to follow up, but I’ll try. Since you seem to know how to effectively manage an engineering team -- or, at the very least, know how it looks like -- please enlighten us: how would you measure productivity/ROI/whatever useful metric?
  - ang_cire 2y ago
    Not them, but I agree with their point that management is, by definition, where the buck stops when it comes to projects, yet are never held accountable when projects go over budget or behind schedule, and that is instead put on developers. I think that productivity has to be defined by deliverables achieved on-time. If one developer is consistently failing to achieve the time boxes that their peers collectively set, that is a red flag for their productivity. If any meaningful *percentage* of your developers aren't hitting their KPIs, that's a management failure. This distinction is important, because you can't accurately measure productivity if you don't accurately set goals. Unfortunately, this assignment of responsibility rarely actually works out properly, and developers get used as scapegoats by bad managers who are in fact themselves unproductive in their NoSP (number of successful projects) metrics, and who go from job to job and group to group blaming bad developer productivity for project failures. There is a reason we hear about developer productivity and not manager productivity, in an industry where most projects fail to meet project goals (which are a management responsibility to ensure, not a developer responsibility).
    - senorrib 2y ago
      Time boxes in software development are a delusion. There's a reason why the entire industry shifted focus from outputs to outcomes when it comes to software engineering.
      - 2y ago
jacknews 2y ago
"turns out" It turns out that 'leadership' are often just a bunch of clueless 'bullies', as in just the top of a dominance hierarchy. Summoned to the CEO's house to discuss the issue, seriously? An obvious attempt to intimidate, imho.
- jimbokun 2y ago
  So CEOs should not ask their direct reports questions about the work their teams are doing?
  - jacknews 2y ago
    "It's the way you tell 'em"
  - karaterobot 2y ago
    The weird power move is that he made his direct report to fly out to his home. The article makes it clear this was not optional.
    - jdlshore 2y ago
      You’re making a lot of assumptions. The reality is that I was going to be in that city for other reasons, and the visit to his house—which I appreciated and wanted—was tacked on to that trip, and followed by a party the CEO hosted for everyone else who was also in town.
      - jacknews 2y ago
        even so, was it optional, or no?
      - karaterobot 2y ago
        Well, okay, I'm not going to contradict your personal experience. But here's what you said in your article: > It started half a year ago, in September 2023. My CEO asked me how I was measuring productivity. I told him it wasn’t possible. He told me I was wrong. I took offense. It got heated. > After things cooled off, he invited me to his house to talk things over in person. (We’re a fully remote company, in different parts of the country, so face time takes some arranging.) I knew I couldn’t just blow off the request, so I decided to approach the question from the standpoint of accountability. How could I demonstrate to my CEO that I was being accountable to the org? So what I knew is that you said you didn't want to do something, you had an argument, he asked you to come to his house, which is not in your home town, to talk things over, and you knew you couldn't just blow off the request, and had to demonstrate your accountability to the organization. True or not, can you see how one might draw the conclusion that you were forced to present yourself to the CEO in his house in order to show that you were going to toe the line?
exabrial 2y ago
I have a simple proposal why SE’s are externally strapped for time: upgrades. No not security patches. Upgrades. Like ‘oh the newest best version of React was released yesterday and I’ve just GOT to have those and management doesn’t understand how productive I’ll be and they shorted us on our last upgrade by not letting us go all in’ This endless treadmill of half baked framework features and ui refreshes streaming in the root problem. I really think a team that can close the gate and seal the seams from the problem would be the most effective. That would mean your most important KPIs would be something like: * number of new features imported from frameworks (close to zero is ideal). These are expensive, like millions of dollars each. * number of security patches applied/response time/etc (maximize these) * eliminate things with no or short LTS, measure weighted lts: (LOC using framework multiplied by framework LTS) Now this in and out itself isn’t sufficient to measure an organization’s productivity, but these would would keep it on the straight and narrow so other measures would become effective.
- mjr00 2y ago
  This is far too narrow of a scope to be broadly applicable. The industry in general does not spend nearly as much time upgrading packages as you imply, and your metrics don't make sense either. How do you measure the number of "features" imported from a framework?
- simonw 2y ago
  I'm with Charity Majors on this one: https://twitter.com/mipsytipsy/status/1778534529298489428 https://twitter.com/mipsytipsy/status/1778534529298489428 "Migrations are not something you can do rarely, or put off, or avoid; not if you are a growing company. Migrations are an ordinary fact of life. Doing them swiftly, efficiently, and -- most of all -- completely is one of the most critical skills you can develop as a team." See also this, one of my favourite essays on software engineering: https://lethain.com/migrations/ https://lethain.com/migrations/
  - Aqueous 2y ago
    "Doing them swiftly, efficiently, and -- most of all -- completely is one of the most critical skills you can develop as a team." That all sounds great. However, I'd like to understand what teams are actually able to do this, because it seems like a complete fantasy. Nobody I've seen is doing migrations swiftly and efficiently. They are giant time-sucks for every company I've ever worked for and any company anyone I know has ever worked for.
nine_zeros 2y ago
Not this again. Maybe one day they should measure the amount of time they spend on this and improve their own productivity by not wasting so much time chasing a fool's errand.
karol 2y ago
Lines of code. Case closed!
- ang_cire 2y ago
  I really hope this is sarcastic. Is a developer that takes 2,000 LoC to fix an issue that another takes 200 to fix 10x more productive, because I'm pretty sure it's the opposite? Productivity has to include "efficiency" somewhere in it, and LoC doesn't capture efficiency at all.
  - khazhoux 2y ago
    Of course if you just take point samples, it fails as you say. But I don't think people really understand what LOC stats look like in practice for engineering teams. The pattern I've consistently seen --across several companies and many teams-- is that most low-performers simply don't commit very much code every month/quarter/year/whatever. I'm not talking about the senior engineers whose job is architectural design and advising and code review, and so on, who contribute a ton to the project without a line of code. I'm talking here only about junior- or mid-level software engineers whose job is supposed to be hands-on-keyboard code. You will always find a number of them who simply push very little code of any significance, compared to their peers. This will be visible in their commits. So while it's true that there's no function productivity=f(LOC), there is still information to be gleaned from reviewing the commits (in detail) of everyone on the team, and often one will see a correlation where the people who are not delivering much value to their team (they fix only few bugs, or they take a very long time to implement things, etc) they more-often-than-not have very low commit stats overall.
    - OJFord 2y ago
      If you insisted on good git hygiene, then I'd have a lot more time for commit count as a productivity metric, and even LoC to a lesser extent.
    - ang_cire 2y ago
      Sure, this is useful for the outliers who just straight don't work as much, but much less useful for telling the difference between a busy person and a productive person. I have seen bad devs who make daily commits with tens or hundreds of LoC, but whose tools never actually convert to production use because they are mired in development hell, until someone pulls the project away from them and hands it to someone else. Devs who are so bad that they legitimately spend weeks trying to make a tool that I then built in 2 days. I think LoC irks me so much, because if I finish someone's "2-month" project in a week, LoC optimization effectively penalizes me for it.
jimbokun 2y ago
I feel like it’s easier to measure for large scale back end systems. Where you can look at things like throughput, scalability, failure rates, latencies, uptime, etc. So to an extent developer productivity can be thought of as the derivative of improvement in those metrics. For front end development, you need to look at user satisfaction measures. Where it’s a bit more difficult to quantify which contributions are most responsible.
- senorrib 2y ago
  I feel like _satisfaction_ is a product measure, not engineering. You can measure the exact same things on frontend and backend.
talkingtab 2y ago
Understanding software development is simple: 1. I want you to build me a borgizaf. How long will it take you and how can I measure your productivity? Now I want you to change the following line of code to print "bye" and answer the same two questions. `console.log("hello");` Any answer to the first question has a certainty of 0%. The answer to the second is almost 100% certain. The certainty is directly dependent on whether you know what you are building AND whether you have direct experience having completed the same tasks. 2. Given some task that is to some extent unknown (see above), how do we proceed? We reduce the uncertainty/degree. This leads directly to "hello world". You start with a program that shows what a borgizaf can do and show it to the people who want it. If it is wrong, evolve it. Iterate. The problem here is that often the people who want the borgizaf can't really tell you what it is. Especially if it has not been made before. But they can tell you what it is not. (See Notes on the Synthesis of Form). Even the people who want a borgizaf may not know what it is. And it may change. 3. In order to increase "productivity" we often have teams of people working on aspects of the same problem. After years of labor each part is done and one day the thing is assembled. Unfortunately, the drive team did not tell anyone they were using front wheel drive, and everyone else thought it was rear wheel drive. Left and right driver position, etc. You don't believe this is possible. One word "Ingres". Iteration must be the whole. Unless you are iterating on the whole, your certainty will always be 100%. Because the pieces might not fit together. 4.The speed at which you develop a borgizaf, is the dependent on the speed of iterations and the degree to which each iteration reduces uncertainty. Really, I think that is all there is to it.
- beretguy 2y ago
  Borgizaf. Nice. I’m borrowing it.
- AnimalMuppet 2y ago
  It's a bit worse than that. The borgizaf is often defined by business development, or somebody like that. They decide they need one after a bunch of customer interviews. But the customers don't actually know what they need. So the business development people are trying to describe something that the users need but don't know that they need, and that doesn't yet exist. Yeah, the main problem is uncertainty, not lines of code to write. You measure progress mostly by how much you reduce uncertainty.
MP_1729 2y ago
Beware Goodheart's Law: "when a measure becomes a target, it ceases to be a good measure". If your goal is stopping to waste time solving bugs, I'm sure you're going to be able to do that. You should have an important counter-metric to see if you're not messing with software. It could be number of reported bugs, crashs in production, etc.
- littlestymaar 2y ago
  Nitpick: It's Goodhart (without the “e”).
- PaulKeeble 2y ago
  Then it becomes the challenger scenario. Various pieces are failing but the whole mission succeeds so everyone ignores the known risks because management is interested in their return on investment. That works right up until the rocket explodes and suddenly there are lots of external people asking serious questions. Boeing is going through the same thing having optimised for ROI as well and its planes are now falling apart on a daily basis. Who always gets in trouble for this? More often than not the developers and operators who in a high pressure environment optimised what they were told to optimise and gamed the metrics a little so they weren't fired or held back in their careers.
- Izkata 2y ago
  Naming it "muda" helps push it that way, too: If any of those higher-ups decide to look up the word, they'll see that you're calling bugfixing "pointless work".
- hinkley 2y ago
  Professional athletes have a lot of telemetry on them. But some of that telemetry makes sense during training, and maybe makes more sense for a brief period of time while they work on technique. You focus on something intensely for a little while, get used to how it feels, then you work on something else. If your other numbers look wrong, or if it's been too long, we look at it again.
deleted 2y ago
[deleted]
qznc 2y ago
First, specify more clearly what you mean by "productivity" to avoid misunderstandings: https://dl.acm.org/doi/10.1145/3454122.3454124 https://dl.acm.org/doi/10.1145/3454122.3454124 It can be about satisfaction & well being, performance, activity, collaboration & communication, or efficiency & flow. These are all different things.
ChrisMarshallNY 2y ago
I'm of the opinion that it isn't really possible to even define "productivity," let alone measure it. Different layers of the org, have different expectations for "productivity," and the problems generally occur, when conflicting definitions collide. I have come to learn that there's really no substitute for hiring really good, experienced engineers, and then manage them with Respect. I know that the SV "Holy Grail" is to come up with The Perfect Process, whereby we can add a monkey, pay them bananas, treat them like circus animals, and come out with perfect code, but I haven't seen anything even close to that, succeed.
karmakaze 2y ago
My TL;DR - estimate (sustainable) Value-add capacity. The current/instantaneous value-add capacity is what the team/department can deliver in a short-time without consideration of hidden costs (tech debt). The 'sustainable' modifier adjusts that to be a long-term average. This is something developers and team managers will certainly know, but is good to see framed from the top-down (though I don't know how many C-level execs know tech debt 'in the gut'). Usually it's ignored until it gets so bad a rewrite is required.
javier_e06 2y ago
The article remind me of an add in Dr. Dobbs Magazine inside back page. It is a Microsoft add with two pictures in it. In the first picture there are two developers in a conference room late at night. There is a Pizza box open with half a pizza and a board with a software design that looks like spaghetti. The developers hair is messy. The looked stress. The second picture is of a group of people walking in with birthday cake on a bright sunny day to the same conference room. The two developers look happy. The manager is somewhere in the picture. I think. Is a MS .Net commercial. I could not find the add in the wayback machine. Companies attach yearly bonus to production/profits meeting goals/targets to increase productivity. Whatever productivity is.
barfbagginus 2y ago
Would it help to demand that your non-technical leadership knows category theory, queuing theory, version control, and infrastructure automation? One way that helps me is that I have nobody on the C suite but me. Because when some lazy business jerk with no knowledge of even a basic functor starts telling me what I should do to be accountable to them, I give them a big fat boot. Obviously a downside is that scaling my organization to more than one head count is going to be a little difficult!
llmblockchain 2y ago
I have a method. Remove your headphones and listen to your teammates. I've always used the "fucks per-hour" as a measurement of productivity. For the uninitiated, the more fucks, the more productivity.. ie, they are looking at the code and trying to do something. True story-- at my first job the fucks were so bad they moved some of us into our own offices.
- scruple 2y ago
  I love this. When they RTO'd us at work, I ended up sitting next to someone who isn't on our team and who isn't an engineer. He complained to his manager that I curse a lot and when they asked my manager to have me tone it down they moved him instead. I'm not being loud or particularly vulgar, just lots of "what the fuck?" and "what is this shit?" and "how the hell does this even work?" being involuntarily mumbled to myself.
  - philk10 2y ago
    https://www.osnews.com/story/19266/wtfsm/ https://www.osnews.com/story/19266/wtfsm/
  - pvdoom 2y ago
    I had the exact opposite experience :( Stating things like "what is this shit"or "This is an absolute dumpster" was taken as demoralising colleagues ...
    - scruple 2y ago
      Oh, well that's pretty funny because most of the time I'm complaining about code I wrote...
OutOfHere 2y ago
A proto-measure is whether the software has active users, whether as a library or application or service, etc., and whether internal or external. That's after a substantial initial seed investment. If it still has no active users, it's time to change focus. It it has active users, then assuming it ultimately rolls up into a revenue generator, it must continue receiving funding. This is not a full measure, but it's a foundation.
asdasdsddd 2y ago
I feel like this doesn't account for the fact that not doing muda makes value-add work take longer
Aurornis 2y ago
Kudos to the author for being honest about the flaw in this metric: > It’s ridiculously easy to cheat this metric. Even if you correctly categorize your muda—it’s very tempting to let edge cases slide—all you have to do is stop fixing bugs, defer some needed upgrades, ignore a security vulnerability... and poof! Happy numbers. At a horrible cost. > Actually, that’s the root of my org’s current capacity problems. They weren’t cheating a metric, but they were under pressure to deliver as much as possible. So they deferred a bunch of maintenance and took some questionable engineering shortcuts. Now they’re paying the price. > Unfortunately, you can get away with cheating this metric for a long time. Years, really. It’s not like you cut quality one month and then the truth comes out the next month. This is a metric that only works when people are scrupulously honest, including with themselves. I've had the same experience with well-meaning productivity metrics collection: Even the execs who were really trying to do the right thing would accidentally invent a new metric that looked fantastic for a couple years, then later collapsed when everything else caught up. By then it might be someone else's job to clean up the mess. Like the author said, this difficulty in this problem is that it can go on for years. If you have an executive team that actually knows how to balance the metrics with the things that are harder to tracked, it might not be a problem. However, as soon as you get an executive looking to game the system for a couple years before jumping to another job, it becomes a doomed venture.
- godelski 2y ago
  I think most people forget that most metrics are easy to hack and all metrics are hackable (yes, even if you use math & science). So the only rational thing to do is be aware and acknowledge the limitations of your metric and be open about that and what context they operate under. If you aren't aware of the underlying assumptions, your metric is borderline useless. It's like the old saying "there's two types of code: those with bugs and those that no one uses." All systems are buggy and if you aren't looking for them you'll get bit. And in great irony, it is why metrics are the downfall of any meritocracy.
  - pixl97 2y ago
    All models are wrong, some models are useful.
yellow_lead 2y ago
> So that’s my productivity measure: value-add capacity. The percentage of engineering time we spend on adding value for users and customers. I think this is an important metric, but it doesn't mean anything if that time is wasted. Imagine the team has no muda, they are spending all their time on value-add, productivity is 100% right? Well no, you still need to succeed at what you're doing, do it well, and in a reasonable time. This isn't a measure of productivity, it's a measure of time allocation.
- ec109685 2y ago
  Agreed, I think. Does his framework include a measure of output too by looking at ROI? So if you are able to deliver more effectively, that should increase ROI independently of whether you reduce the amount of muda time.
- ozim 2y ago
  If I code up to the specification given by the business, a feature that annoys customers and makes them switch to the competition, is it my problem? For engineering department amount of "value-add" or how many features they deliver (to the specification of course, if the requirements are bad, it is also on business) is the only measure of productivity. Because that is something they can have control over. Reasonable time is implied by delivering amount of value-add. Don't go overboard with the definition, it should be context of engineering, not the whole company. If business cannot come up with good features that time is wasted for the company but engineering is doing what they are paid for.
Izkata 2y ago
> Like any engineering organization, we spend some percent of our time on fixing bugs, performing maintenance, and other things that are necessary but don’t add value from a customer or user perspective. The Japanese term for this is muda. That translates as "pointless" or "useless"... that's like, the opposite of "necessary". Just sticking with an English word like "foundation" would make sense here - necessary, but not something a customer or user typically sees. Even further, metaphorically, fixing bugs = stabilizing the foundation. Using foreign words just to sound fancy and ending up with the wrong meaning is just annoying.
- jlawson 2y ago
  Alternatives: foundation, plumbing, substructure, support work, support structure, backend, sustainability, backstage, logistics
  - myhf 2y ago
    "necessary but not sufficient"
- kitten_mittens_ 2y ago
  A good chunk of the English lexicon was foreign at one point or another: https://en.wikipedia.org/wiki/Inkhorn_term https://en.wikipedia.org/wiki/Inkhorn_term
- jessetemp 2y ago
  They're saying the things that don't add value are muda (waste), not the foundational stuff. I only know that because my employer uses the same jargon. And since almost no one speaks phonetically-spelled-out Japanese, they always follow it with (waste) in parentheses
- jdlshore 2y ago
  “Muda” is a technical term with a specific meaning. (Google it.) It originated in the Toyota Production System, then migrated to software development via the Poppendiecks’ “Lean Software Development.” It doesn’t mean “pointless” any more than legacy software means “an inheritance.”
dboreham 2y ago
The executive summary of this article (for me) is: burst the delusion bubble around your stake holders. This is admirable, but that delusion existed in the first place for a reason. The reason still exists.
a_c 2y ago
I might have skimmed too fast. Productivity seemed not defined? IMO productivity is all about making product that get used. And profitability is how much people willing to pay to use such product.
neilv 2y ago
I wonder how well the engineering/product would've gone, in that case, without that productivity-measuring flavor of leadership being asserted.
w10-1 2y ago
Risk is lost in the discussion but very relevant to management. In theory, in the most productive engineering organization, each person is the expert at what they're doing and work is factored to avoid overlap. This happens somewhat naturally as people silo (to avoid conflict and preserve flexibility). This is actually the perfect system, if your people are perfect: their incentives are aligned, they're working with goodwill, etc. But that makes every person critical, few people can operate at that level all the time, and fewer still want to work alone doing the same thing ad infinitum. Also, needs change, re-factors work and its mapping to workers. So then you expand the scope to productivity over time with accommodation for change - i.e., capacity. But by definition, capacity far exceeds output, and is thus unobservable and unmeasurable. So instead you experiment and measure: switch roles, re-organize work, address new kinds of problems and prototype new kinds of solutions, bounded by reasonably anticipated product needs. And to inject this uncertainty, you actually have to inject a lot of countervailing positivity, to deflect the fear of measurement or failure, to encourage sharing, etc. Unfortunately, these experiments and freedom from consequences is pretty indistinguishable from the bureaucratic games and padding that productivity measures are designed to limit.
deleted 2y ago
[deleted]
OJFord 2y ago
With articles like this I always think it'd be really interesting to hear from others on the team. I find it really hard to imagine there's not going to be pressure (either self-inflicted or from above) to 'game' it, or how do you decide what's gaming and what's pragmatic anyway - perhaps better said I find it hard to imagine people don't feel like it's being gamed, that they're having to add value where they think they should be maintaining, even if OP or whoever deciding that thinks it's the right call. (And to be clear I can't say and am not saying which is right, from over here with zero information about any such decisions.) It could be fun to have a platform for honest blogs about work stuff where it's grouped by people verified to work (or have worked) at the same place. (Or, I realise as I write that, a descent into in-fighting and unprofessionalism..)
mdgrech23 2y ago
"but the ceo had a scheduling conflict so he couldn't come to this thing that was really important to him". Does that ever make sense to anyone? It never made sense to me. I think it's about classism. These guys only want to hang out w/ the lower class if they're at the front of the room talking otherwise they want to come off as mysterious and too busy to have any time. What a load of shit.
- rhelz 2y ago
  cf. with the observation that "leadership teams like accountability a lot more when its other people being held accountable."
booleandilemma 2y ago
There's no shortcut to paying attention. That is, paying attention to who's doing the real work. Any metric can be gamed.
ozim 2y ago
I would point out that Martin Fowler, Kent Beck and Gergely Orosz were writing that you cannot measure productivity of a single team member. *You can get a rough sense of a team's output by looking at how many features they deliver per iteration. It's a crude sense, but you can get a sense of whether a team's speeding up, or a rough sense if one team is more productive than another.* So in that sense "value add time" will be a valid metric. But it is still not a single number that one can give without context to the CEO, that CEO still has to ask questions like "we usually have X time spent on fixing bugs, aren't you gaming the system to get your bonus?". That is the CEO job to understand the metric not to be gamed...
deleted 2y ago
[deleted]
jlas 2y ago
Don't overlook sources of anti-productivity which might be easier to measure like the time it takes to build & test code changes.
danielovichdk 2y ago
Productivity is a totally irrelevant metric. It takes 2 people, 5 years to built a 50 foot sailboat. It takes 10 people, 4 months. Which is most productive? That depends on the price and the quality of course. Never measure for quantity. Always measure for quality in the environment you are competing in. And since most professional work is about making money/use value, it would be a could start to measure how much money/use value people are contributing to. If people are not actively contributing to this they are not productive.
siliconc0w 2y ago
I like DORA metrics as a guideline (Deployment frequency, lead time for changes, change failure rate, time to restore) which aren't perfect but are fairly straight forward to understand and measure. I might look at number of production touches(manual interventions in the delivery pipeline), story points estimated, story points delivered, number of pages, number of bugs created/closed/closed w/ change, number of errors/exceptions. It's possible to hide technical debt here but it eventually start to manifest in one of the stability metrics (bugs, exception, production touches, etc) or a drop in velocity . Though is more about engineering delivery of the product roadmap. I do like the 'bets' framework in the OP to figuring out whether the product roadmap actually results in meeting the desired goals for the organization (growth, profits, whatever) but these should be measured as different things.
rhelz 2y ago
I'm reminded of a conversation I had with a colleague from Georgia, who had a beautiful southern drawl. He said, "The way Ah measure Ahr Oh Ah is Ahr over Ah." Assigning productivity numbers to individual engineers is enormously difficult-- but at the level of VP of all programming, it seems simple: At the end of a quarter, you know how much you spent on programmers during that quarter. And a couple of quarters after that, you should know how much value was added. Why not calculate ROI as R over I???
WatchDog 2y ago
It seems like the main problem with any productivity metric, is that once you tell your workers what the metric is, it will be gamed. However if you collect the metrics in secret, and only use them as a guide for where productivity could be improved, it seems they could be more useful. I wonder if any engineering managers are doing something like that? I guess it would be hard to scale to a large organization, as the you would need to share the metrics with the line managers.
- senkora 2y ago
  I suspect that many quietly effective employees are doing something like that, and that it is a smart way to increase the performance of yourself and your team. I also suspect that it is not possible to share the metrics, and that you would need to do "parallel construction" to justify your decisions without revealing your real methodology.
Willish42 2y ago
> Actually, that’s the root of my org’s current capacity problems. They weren’t cheating a metric, but they were under pressure to deliver as much as possible. So they deferred a bunch of maintenance and took some questionable engineering shortcuts. Now they’re paying the price. > Unfortunately, you can get away with cheating this metric for a long time. Years, really. It’s not like you cut quality one month and then the truth comes out the next month. This is a metric that only works when people are scrupulously honest, including with themselves. I fear this is what's happening at basically every major tech firm in the world right now. We might not fully realize the stupid breakages and tech debt of these decisions (namely: layoffs and large refactors of the product and system to inject AI everywhere) for _quite_ some time. I fear it'll be just like "social first" and "mobile first" iterations of the web that create a bunch of cruft to be cleaned up later.
matchagaucho 2y ago
The best companies can churn their metrics quarterly. Early days of FB were often driven by MAU. Then P95 performance became a priority, leading to higher DAU.
cess11 2y ago
If a boss came to me and said 'hey from now on we're going to describe your work in terms of gambling' whatever loyalty I had would have collapsed.
aryehof 2y ago
It seems to me that “productivity” is being used here carelessly. It is a ratio of effort and resulting output. High productivity will produce more output given a specific effort, than low productivity, given that other factors like quality are constant. Frankly, in software we cant even determine the extent and relationships between requirements, effort, quality and output in any predictable way. Given this, isn’t an attempt to produce some resulting measure like productivity meaningless?
thefaux 2y ago
All of this feels like a side effect of a low trust environment. I respect the author's creativity, but the environment described sounds highly dysfunctional. In my opinion, if the ceo is demanding an arbitrary metric from his subordinate without being able to articulate what they are looking for sufficiently that the subordinate has to invent a made up number just to justify their existence, that sounds like poor leadership to me. Of course, this is only based on the author's side and perhaps I'd feel differently if I heard the other side.
- bennyelv 2y ago
  It is (source: I work in an org like this), but your aim as a leader, the VP in this case, is to do whatever is necessary to enable your team to succeed, given the constraints that exist. Ineffective company operation and other people’s hangups and shortcomings are constraints. Does the nirvana where you are never limited by the ineffective operation of others exist anywhere? Some companies are better, some are worse, but all of them are collections of people with limitations at the end of the day!
la64710 2y ago
TLDR This is an article about a new way to measure software engineering productivity. It discusses the challenges of measuring productivity in software engineering. The author proposes a metric called value-add capacity. This metric focuses on the percentage of time engineers spend on activities that directly add value to the customer. The author acknowledges that this metric can be easily cheated. However, he believes it can be useful if used honestly.
dusted 2y ago
I misread the title as the more interesting "Is productivity a useful measure?"
jerrygoyal 2y ago
how to measure dev productivity? you can measure it by how many times a dev rewrites the code before the code gets merged.

249 points by rglullis 2y ago

> In other words, in the absence of RoI measures, the percent of engineering time spent on value-add activities is a pretty good proxy for productivity. What about using self-rated satisfaction of your workers in terms of how happy they are with their engineering solutions? I find that when I solve a really nice problem and add something that is cool, or fix a very annoying bug, I feel very satisfied with the result. Whereas, I have produced things in the past (when I worked as a programmer) that seemed to make management happy that I wasn't satisfied with, and it seemed to me at least that the results weren't as useful in the end.

yen223 2y ago
To put it bluntly, Goodhart's law will really kick your ass if you went with that.
- deleted 2y ago
  [deleted]
mjr00 2y ago
This is way too subjective; I've met way too many developers throughout my career who loved to spend their time crafting "perfect" code rather than deliver customer-facing value for this to work as a measure that aligns with business objectives.
jimbokun 2y ago
Useful to know, but different from productivity.

Archelaos 2y ago

> * The value it’s estimated to generate > * The amount we’re willing to bet A reasonable bet should correspond to the (estimated) probability of its outcome. So the amount someone is willing to bet should directly reflect the value it is estimated to generate. Betting higher or lower makes no sense.

datadrivenangel 2y ago
It depends on your risk tolerance! We may want to bet more for more likely outcomes, even if the expected return is lower, if we value predictability and the cost of a bad portfolio outcome is high.

lifeisstillgood 2y ago

I think there are two answers here. The first is treating software develolment as a process - and using some form of statistical process control to “manage” the process - reduce waste etc. this is it turns out exactly what Agile / Scrum was / is all about. Remember those “retrospectives” we are supposed to do - that’s the point at which we whip out a diagram and see if we are trending out of band. It’s fine. It works really and it does help reduce waste - which for any sane world reduced waste is a proxy for productivity. But no matter what that is only “doing things right”. It is not “doing the right thing”. At this point we can also say we have tactical and operational parts under control (or at least monitored). The “doing the right thing” (or strategy) is next and this means something most business refuse to accept - that their grand strategy, their Vision, might be total bollocks. Searching for productivity in software teams is the new version of blaming lazy workers in the factory floor - we are all software ompanies now, and if you are not succeeding you can either blame your crappy product market fit, your strategy or you can whip the employees harder. Guess which is easier. So if you and the CEO need to make fucking appointments to have a honest chat about strategy something has gone terribly wrong with your companies ability to quickly respond strategically - if you have a problem, it’s not the employees that need to run faster. edit: yeah I started ranting there, sorry. The whole “takes ages to get face to face time” is a red flag, but you can get around it. As for the rest, I really respect the attempt to treat the whole company as a system - not just the software team - the product bet idea clearly sets out goals and places emphasis on whole org. That’s really great but as you say “executives prefer if other people are held accountable”. But if they can learn a company can be programmable (then they might learn to start programming)

robertlagrant 2y ago
> this means something most business refuse to accept - that their grand strategy, their Vision, might be total bollocks Actually this is often talked about, but only internally to the senior team. You generally don't want to be saying "actually widgets might be a bad idea" which might horribly demoralise your widget factory workers, when you might well conclude after discussion and analysis that actually they're still a good idea.

didgeoridoo 2y ago

The “fatal flaw” of skipping maintenance to max out value-add work seems like it could be addressed by ensuring that any accumulated tech debt be properly accounted for, rather than swept under the carpet. You’d consider “productivity” then to be value-add work minus identified tech debt. Since calling out tech debt would hurt leadership’s own metrics, you’d need a mechanism to allow individual engineers to blamelessly (and probably anonymously) report, validate, and attach a cost metric to this tech debt. The org would then be incentivized to balance work that delivers immediate value with work that represents an investment in making the team more efficient and effective at delivering that value in the long run.

datadrivenangel 2y ago
That's the thing though! Quality :tm: is very hard to account for, especially when the quality is in the system, tooling, and process to build the quality end product. And often times technical debt isn't actually something you can put on the balance sheet or bug tracker. It's all the little investments in the future that are deferred or skipped. That one code change is so minor you can phone in the review, you'll write better documentation for that new feature in a few weeks when you have some time, etc.
- didgeoridoo 2y ago
  It’s a good practice to note things for later and why they should be done though, right? Even if it is never intended to be worked on, noting that you would write more docs for this class but you don’t have time is an important indicator of productive capacity for leadership. If I start seeing a lot of that as an executive, I should start to worry if we’re building our value-add on a foundation of sand.
- duderific 2y ago
  > you'll write better documentation for that new feature in a few weeks when you have some time The trick here is to make a ticket for it. At least then, it can be prioritized appropriately, instead of disappearing into a void.

roenxi 2y ago

I really like this article so I am sad to be cynical about it. Nevertheless. If you're engaging in C-suite politics as described in this article, what on earth would be the motivation for writing all this in public? I doubt the situation will evolve favourably to Mr Shore if the leadership team starts reading this. Full agreement with the perspective though. CEOs (& consultant friends) generally have some extremely reasonable expectations about software engineer productivity based on experiences in manufacturing, primary industries, healthcare, service and public sectors. Surely we need accountability and productivity metrics! It is a long and painful process that may never end watching them gathering the evidence needed to learn that software engineer productivity is different. A sad history repeating over and over again like clockwork. This CTO seems to be finding a golden path for how to manage expectations without beating that over people's heads and I'm taking notes. I don't think it'll work out exactly the way he hopes but I love the attempt at making his team productive.

simonw 2y ago
If I was a CEO with a VP who wrote something this realistic and honest, my response would be to pressure the rest of my execs to give me that same level of quality information. Breath of fresh air!
AtlasBarfed 2y ago
One of my long standing beefs with C-suites and their MBAs and whatever meat factory grinds them out is the continued lack of comprehension of basic software and IT management. Back in the early 2000s, this could be saliently characterized by the exuberance many CEOs of companies bragged about how IT-ignorant they were. I saw the "my secretary prints out my emails" written proudly in print many a time in the CEO worship magazines like Fortune and the like. Every decade of continued IT penetration and performance benefits in business are continuing the principle: the health of your business is strongly tied to its IT health. BUT MY COMPANIES JUST STAMPS WIDGETS. Ok, look. MBAs long ago accepted that every business, regardless of what it does, needs two things: accounting and finance. So every MBA gets a decent education on those. Well, hate to tell you, MBAs need to know the fundamentals of IT. Basically, C-suites not understanding productivity measures in IT are hard is stupid not only because it is a basic aspect of IT management, but its a bit more disturbing than that: YOU CAN'T MEASURE MANAGER PRODUCTIVITY EITHER. How does traditional MBA measure middle managers and those types of orgs? That is rife with all the problems of measuring developers, not the least of it being the "juke the stats" universal phenomenon. That's why the most important single indicator of manager ability is: headcount. Headcount headcount headcount. The second is size of budget: bigger bigger bigger. Of course everyone that has working in companies knows the hilarity of what that produces: bloated orgs, pointless spending to use up a budget otherwise it gets cut, building empires, etc. The essence of developer that C-suites need to understand is that they are not factory workers: they are in fact MANAGERs. They just don't manage people, they manage virtual employees/subordinates (software). Go ahead, ask any software dev in an org what they do. Generally, somewhere in there is a bunch of acronyms and the fact that they ... wait for it ... MANAGE those acronyms. Krazam's microservices is a perfect example of this. What is that long diatribe essentially about? MANAGING all those services. That gets to another major undercurrent of IT. Managers want to place IT employees in the "worker bee" category. They don't want IT workers to be managers, and entitled to the great benefits according the management class. The long running IT salary advantage in the US, and the constant ebb and flow between management and IT "working" to push that down is a major social undercurrent here.

thcipriani 2y ago

Seems like the author of this is aware of Goodhart's law, but it bears repeating: both of the proposed real measures (RoI and value-add capacity) could come at the expense of uptime and stability. John Doerr talks about balancing OKRs, so you could have you product bets and...something about stability: hitting SLAs, whatever. And the term "product bets" makes the leadership team sound like Mortimer and Randolph Duke.

haskellandchill 2y ago

What about the Bayesian methods shown in "How to Measure Anything"? They have been applied to Cybersecurity ("How to Measure Anything in Cybersecurity Risk" in a very thorough and convincing manner. It looks like the business around it is trying to apply it to product management (https://hubbardresearch.com/shop/measure-anything-project-management-seminar-plus/ https://hubbardresearch.com/shop/measure-anything-project-ma...). Basically the idea is when things are hard to measure we should not abandon quantitative scales and use qualitative ones (like t-shirt sizes) but instead use probabilities to quantify our uncertainty and leverage techniques like bayesian updates, confidence intervals, and monte carlo simulations.

apwheele 2y ago
This is not inconsistent with How to Measure Anything IMO (I like that book as well). The biggest issue to me is that he does not define actual follow ups on ROI -- it is all estimated in this framework. So it is all good to define how to prioritize, it is not helpful though retrospectively to see if people are making good estimates. My work very rarely is a nicely isolated new thing -- I am building something on-top of an already existing product. In these scenarios ROI is more difficult -- you need some sort of counterfactual profit absent the upgrade. Most people just take total profit for some line, which is very misleading in the case of incremental improvements.
- haskellandchill 2y ago
  The problem is the muda should have expected values associated with them. Bugs and security vulnerabilities do cost money, these are 90% confidence intervals of dollar impact from How to Measure.
dentemple 2y ago
Do you have any links to back up your information that doesn't cost $17,250 to view?
- haskellandchill 2y ago

ltbarcly3 2y ago

[flagged]

senorrib 2y ago
This comment is so incredibly useless that it’s hard to follow up, but I’ll try. Since you seem to know how to effectively manage an engineering team -- or, at the very least, know how it looks like -- please enlighten us: how would you measure productivity/ROI/whatever useful metric?
- ang_cire 2y ago
  Not them, but I agree with their point that management is, by definition, where the buck stops when it comes to projects, yet are never held accountable when projects go over budget or behind schedule, and that is instead put on developers. I think that productivity has to be defined by deliverables achieved on-time. If one developer is consistently failing to achieve the time boxes that their peers collectively set, that is a red flag for their productivity. If any meaningful *percentage* of your developers aren't hitting their KPIs, that's a management failure. This distinction is important, because you can't accurately measure productivity if you don't accurately set goals. Unfortunately, this assignment of responsibility rarely actually works out properly, and developers get used as scapegoats by bad managers who are in fact themselves unproductive in their NoSP (number of successful projects) metrics, and who go from job to job and group to group blaming bad developer productivity for project failures. There is a reason we hear about developer productivity and not manager productivity, in an industry where most projects fail to meet project goals (which are a management responsibility to ensure, not a developer responsibility).
  - senorrib 2y ago
    Time boxes in software development are a delusion. There's a reason why the entire industry shifted focus from outputs to outcomes when it comes to software engineering.
    - 2y ago

jacknews 2y ago

"turns out" It turns out that 'leadership' are often just a bunch of clueless 'bullies', as in just the top of a dominance hierarchy. Summoned to the CEO's house to discuss the issue, seriously? An obvious attempt to intimidate, imho.

jimbokun 2y ago
So CEOs should not ask their direct reports questions about the work their teams are doing?
- jacknews 2y ago
  "It's the way you tell 'em"
- karaterobot 2y ago
  The weird power move is that he made his direct report to fly out to his home. The article makes it clear this was not optional.
  - jdlshore 2y ago
    You’re making a lot of assumptions. The reality is that I was going to be in that city for other reasons, and the visit to his house—which I appreciated and wanted—was tacked on to that trip, and followed by a party the CEO hosted for everyone else who was also in town.
    - jacknews 2y ago
      even so, was it optional, or no?
    - karaterobot 2y ago
      Well, okay, I'm not going to contradict your personal experience. But here's what you said in your article: > It started half a year ago, in September 2023. My CEO asked me how I was measuring productivity. I told him it wasn’t possible. He told me I was wrong. I took offense. It got heated. > After things cooled off, he invited me to his house to talk things over in person. (We’re a fully remote company, in different parts of the country, so face time takes some arranging.) I knew I couldn’t just blow off the request, so I decided to approach the question from the standpoint of accountability. How could I demonstrate to my CEO that I was being accountable to the org? So what I knew is that you said you didn't want to do something, you had an argument, he asked you to come to his house, which is not in your home town, to talk things over, and you knew you couldn't just blow off the request, and had to demonstrate your accountability to the organization. True or not, can you see how one might draw the conclusion that you were forced to present yourself to the CEO in his house in order to show that you were going to toe the line?

exabrial 2y ago

I have a simple proposal why SE’s are externally strapped for time: upgrades. No not security patches. Upgrades. Like ‘oh the newest best version of React was released yesterday and I’ve just GOT to have those and management doesn’t understand how productive I’ll be and they shorted us on our last upgrade by not letting us go all in’ This endless treadmill of half baked framework features and ui refreshes streaming in the root problem. I really think a team that can close the gate and seal the seams from the problem would be the most effective. That would mean your most important KPIs would be something like: * number of new features imported from frameworks (close to zero is ideal). These are expensive, like millions of dollars each. * number of security patches applied/response time/etc (maximize these) * eliminate things with no or short LTS, measure weighted lts: (LOC using framework multiplied by framework LTS) Now this in and out itself isn’t sufficient to measure an organization’s productivity, but these would would keep it on the straight and narrow so other measures would become effective.

mjr00 2y ago
This is far too narrow of a scope to be broadly applicable. The industry in general does not spend nearly as much time upgrading packages as you imply, and your metrics don't make sense either. How do you measure the number of "features" imported from a framework?
simonw 2y ago
I'm with Charity Majors on this one: https://twitter.com/mipsytipsy/status/1778534529298489428 https://twitter.com/mipsytipsy/status/1778534529298489428 "Migrations are not something you can do rarely, or put off, or avoid; not if you are a growing company. Migrations are an ordinary fact of life. Doing them swiftly, efficiently, and -- most of all -- completely is one of the most critical skills you can develop as a team." See also this, one of my favourite essays on software engineering: https://lethain.com/migrations/ https://lethain.com/migrations/
- Aqueous 2y ago
  "Doing them swiftly, efficiently, and -- most of all -- completely is one of the most critical skills you can develop as a team." That all sounds great. However, I'd like to understand what teams are actually able to do this, because it seems like a complete fantasy. Nobody I've seen is doing migrations swiftly and efficiently. They are giant time-sucks for every company I've ever worked for and any company anyone I know has ever worked for.

nine_zeros 2y ago

Not this again. Maybe one day they should measure the amount of time they spend on this and improve their own productivity by not wasting so much time chasing a fool's errand.

karol 2y ago

Lines of code. Case closed!

ang_cire 2y ago
I really hope this is sarcastic. Is a developer that takes 2,000 LoC to fix an issue that another takes 200 to fix 10x more productive, because I'm pretty sure it's the opposite? Productivity has to include "efficiency" somewhere in it, and LoC doesn't capture efficiency at all.
- khazhoux 2y ago
  Of course if you just take point samples, it fails as you say. But I don't think people really understand what LOC stats look like in practice for engineering teams. The pattern I've consistently seen --across several companies and many teams-- is that most low-performers simply don't commit very much code every month/quarter/year/whatever. I'm not talking about the senior engineers whose job is architectural design and advising and code review, and so on, who contribute a ton to the project without a line of code. I'm talking here only about junior- or mid-level software engineers whose job is supposed to be hands-on-keyboard code. You will always find a number of them who simply push very little code of any significance, compared to their peers. This will be visible in their commits. So while it's true that there's no function productivity=f(LOC), there is still information to be gleaned from reviewing the commits (in detail) of everyone on the team, and often one will see a correlation where the people who are not delivering much value to their team (they fix only few bugs, or they take a very long time to implement things, etc) they more-often-than-not have very low commit stats overall.
  - OJFord 2y ago
    If you insisted on good git hygiene, then I'd have a lot more time for commit count as a productivity metric, and even LoC to a lesser extent.
  - ang_cire 2y ago
    Sure, this is useful for the outliers who just straight don't work as much, but much less useful for telling the difference between a busy person and a productive person. I have seen bad devs who make daily commits with tens or hundreds of LoC, but whose tools never actually convert to production use because they are mired in development hell, until someone pulls the project away from them and hands it to someone else. Devs who are so bad that they legitimately spend weeks trying to make a tool that I then built in 2 days. I think LoC irks me so much, because if I finish someone's "2-month" project in a week, LoC optimization effectively penalizes me for it.

jimbokun 2y ago

I feel like it’s easier to measure for large scale back end systems. Where you can look at things like throughput, scalability, failure rates, latencies, uptime, etc. So to an extent developer productivity can be thought of as the derivative of improvement in those metrics. For front end development, you need to look at user satisfaction measures. Where it’s a bit more difficult to quantify which contributions are most responsible.

senorrib 2y ago
I feel like _satisfaction_ is a product measure, not engineering. You can measure the exact same things on frontend and backend.

talkingtab 2y ago

Understanding software development is simple: 1. I want you to build me a borgizaf. How long will it take you and how can I measure your productivity? Now I want you to change the following line of code to print "bye" and answer the same two questions. `console.log("hello");` Any answer to the first question has a certainty of 0%. The answer to the second is almost 100% certain. The certainty is directly dependent on whether you know what you are building AND whether you have direct experience having completed the same tasks. 2. Given some task that is to some extent unknown (see above), how do we proceed? We reduce the uncertainty/degree. This leads directly to "hello world". You start with a program that shows what a borgizaf can do and show it to the people who want it. If it is wrong, evolve it. Iterate. The problem here is that often the people who want the borgizaf can't really tell you what it is. Especially if it has not been made before. But they can tell you what it is not. (See Notes on the Synthesis of Form). Even the people who want a borgizaf may not know what it is. And it may change. 3. In order to increase "productivity" we often have teams of people working on aspects of the same problem. After years of labor each part is done and one day the thing is assembled. Unfortunately, the drive team did not tell anyone they were using front wheel drive, and everyone else thought it was rear wheel drive. Left and right driver position, etc. You don't believe this is possible. One word "Ingres". Iteration must be the whole. Unless you are iterating on the whole, your certainty will always be 100%. Because the pieces might not fit together. 4.The speed at which you develop a borgizaf, is the dependent on the speed of iterations and the degree to which each iteration reduces uncertainty. Really, I think that is all there is to it.

beretguy 2y ago
Borgizaf. Nice. I’m borrowing it.
AnimalMuppet 2y ago
It's a bit worse than that. The borgizaf is often defined by business development, or somebody like that. They decide they need one after a bunch of customer interviews. But the customers don't actually know what they need. So the business development people are trying to describe something that the users need but don't know that they need, and that doesn't yet exist. Yeah, the main problem is uncertainty, not lines of code to write. You measure progress mostly by how much you reduce uncertainty.

MP_1729 2y ago

Beware Goodheart's Law: "when a measure becomes a target, it ceases to be a good measure". If your goal is stopping to waste time solving bugs, I'm sure you're going to be able to do that. You should have an important counter-metric to see if you're not messing with software. It could be number of reported bugs, crashs in production, etc.

littlestymaar 2y ago
Nitpick: It's Goodhart (without the “e”).
PaulKeeble 2y ago
Then it becomes the challenger scenario. Various pieces are failing but the whole mission succeeds so everyone ignores the known risks because management is interested in their return on investment. That works right up until the rocket explodes and suddenly there are lots of external people asking serious questions. Boeing is going through the same thing having optimised for ROI as well and its planes are now falling apart on a daily basis. Who always gets in trouble for this? More often than not the developers and operators who in a high pressure environment optimised what they were told to optimise and gamed the metrics a little so they weren't fired or held back in their careers.
Izkata 2y ago
Naming it "muda" helps push it that way, too: If any of those higher-ups decide to look up the word, they'll see that you're calling bugfixing "pointless work".
hinkley 2y ago
Professional athletes have a lot of telemetry on them. But some of that telemetry makes sense during training, and maybe makes more sense for a brief period of time while they work on technique. You focus on something intensely for a little while, get used to how it feels, then you work on something else. If your other numbers look wrong, or if it's been too long, we look at it again.

deleted 2y ago

[deleted]

qznc 2y ago

First, specify more clearly what you mean by "productivity" to avoid misunderstandings: https://dl.acm.org/doi/10.1145/3454122.3454124 https://dl.acm.org/doi/10.1145/3454122.3454124 It can be about satisfaction & well being, performance, activity, collaboration & communication, or efficiency & flow. These are all different things.

ChrisMarshallNY 2y ago

I'm of the opinion that it isn't really possible to even define "productivity," let alone measure it. Different layers of the org, have different expectations for "productivity," and the problems generally occur, when conflicting definitions collide. I have come to learn that there's really no substitute for hiring really good, experienced engineers, and then manage them with Respect. I know that the SV "Holy Grail" is to come up with The Perfect Process, whereby we can add a monkey, pay them bananas, treat them like circus animals, and come out with perfect code, but I haven't seen anything even close to that, succeed.

karmakaze 2y ago

My TL;DR - estimate (sustainable) Value-add capacity. The current/instantaneous value-add capacity is what the team/department can deliver in a short-time without consideration of hidden costs (tech debt). The 'sustainable' modifier adjusts that to be a long-term average. This is something developers and team managers will certainly know, but is good to see framed from the top-down (though I don't know how many C-level execs know tech debt 'in the gut'). Usually it's ignored until it gets so bad a rewrite is required.

javier_e06 2y ago

The article remind me of an add in Dr. Dobbs Magazine inside back page. It is a Microsoft add with two pictures in it. In the first picture there are two developers in a conference room late at night. There is a Pizza box open with half a pizza and a board with a software design that looks like spaghetti. The developers hair is messy. The looked stress. The second picture is of a group of people walking in with birthday cake on a bright sunny day to the same conference room. The two developers look happy. The manager is somewhere in the picture. I think. Is a MS .Net commercial. I could not find the add in the wayback machine. Companies attach yearly bonus to production/profits meeting goals/targets to increase productivity. Whatever productivity is.

barfbagginus 2y ago

Would it help to demand that your non-technical leadership knows category theory, queuing theory, version control, and infrastructure automation? One way that helps me is that I have nobody on the C suite but me. Because when some lazy business jerk with no knowledge of even a basic functor starts telling me what I should do to be accountable to them, I give them a big fat boot. Obviously a downside is that scaling my organization to more than one head count is going to be a little difficult!

llmblockchain 2y ago

I have a method. Remove your headphones and listen to your teammates. I've always used the "fucks per-hour" as a measurement of productivity. For the uninitiated, the more fucks, the more productivity.. ie, they are looking at the code and trying to do something. True story-- at my first job the fucks were so bad they moved some of us into our own offices.

scruple 2y ago
I love this. When they RTO'd us at work, I ended up sitting next to someone who isn't on our team and who isn't an engineer. He complained to his manager that I curse a lot and when they asked my manager to have me tone it down they moved him instead. I'm not being loud or particularly vulgar, just lots of "what the fuck?" and "what is this shit?" and "how the hell does this even work?" being involuntarily mumbled to myself.
- philk10 2y ago
  https://www.osnews.com/story/19266/wtfsm/ https://www.osnews.com/story/19266/wtfsm/
- pvdoom 2y ago
  I had the exact opposite experience :( Stating things like "what is this shit"or "This is an absolute dumpster" was taken as demoralising colleagues ...
  - scruple 2y ago
    Oh, well that's pretty funny because most of the time I'm complaining about code I wrote...

OutOfHere 2y ago

A proto-measure is whether the software has active users, whether as a library or application or service, etc., and whether internal or external. That's after a substantial initial seed investment. If it still has no active users, it's time to change focus. It it has active users, then assuming it ultimately rolls up into a revenue generator, it must continue receiving funding. This is not a full measure, but it's a foundation.

asdasdsddd 2y ago

I feel like this doesn't account for the fact that not doing muda makes value-add work take longer

Aurornis 2y ago

Kudos to the author for being honest about the flaw in this metric: > It’s ridiculously easy to cheat this metric. Even if you correctly categorize your muda—it’s very tempting to let edge cases slide—all you have to do is stop fixing bugs, defer some needed upgrades, ignore a security vulnerability... and poof! Happy numbers. At a horrible cost. > Actually, that’s the root of my org’s current capacity problems. They weren’t cheating a metric, but they were under pressure to deliver as much as possible. So they deferred a bunch of maintenance and took some questionable engineering shortcuts. Now they’re paying the price. > Unfortunately, you can get away with cheating this metric for a long time. Years, really. It’s not like you cut quality one month and then the truth comes out the next month. This is a metric that only works when people are scrupulously honest, including with themselves. I've had the same experience with well-meaning productivity metrics collection: Even the execs who were really trying to do the right thing would accidentally invent a new metric that looked fantastic for a couple years, then later collapsed when everything else caught up. By then it might be someone else's job to clean up the mess. Like the author said, this difficulty in this problem is that it can go on for years. If you have an executive team that actually knows how to balance the metrics with the things that are harder to tracked, it might not be a problem. However, as soon as you get an executive looking to game the system for a couple years before jumping to another job, it becomes a doomed venture.

godelski 2y ago
I think most people forget that most metrics are easy to hack and all metrics are hackable (yes, even if you use math & science). So the only rational thing to do is be aware and acknowledge the limitations of your metric and be open about that and what context they operate under. If you aren't aware of the underlying assumptions, your metric is borderline useless. It's like the old saying "there's two types of code: those with bugs and those that no one uses." All systems are buggy and if you aren't looking for them you'll get bit. And in great irony, it is why metrics are the downfall of any meritocracy.
- pixl97 2y ago
  All models are wrong, some models are useful.

yellow_lead 2y ago

> So that’s my productivity measure: value-add capacity. The percentage of engineering time we spend on adding value for users and customers. I think this is an important metric, but it doesn't mean anything if that time is wasted. Imagine the team has no muda, they are spending all their time on value-add, productivity is 100% right? Well no, you still need to succeed at what you're doing, do it well, and in a reasonable time. This isn't a measure of productivity, it's a measure of time allocation.

ec109685 2y ago
Agreed, I think. Does his framework include a measure of output too by looking at ROI? So if you are able to deliver more effectively, that should increase ROI independently of whether you reduce the amount of muda time.
ozim 2y ago
If I code up to the specification given by the business, a feature that annoys customers and makes them switch to the competition, is it my problem? For engineering department amount of "value-add" or how many features they deliver (to the specification of course, if the requirements are bad, it is also on business) is the only measure of productivity. Because that is something they can have control over. Reasonable time is implied by delivering amount of value-add. Don't go overboard with the definition, it should be context of engineering, not the whole company. If business cannot come up with good features that time is wasted for the company but engineering is doing what they are paid for.

Izkata 2y ago

> Like any engineering organization, we spend some percent of our time on fixing bugs, performing maintenance, and other things that are necessary but don’t add value from a customer or user perspective. The Japanese term for this is muda. That translates as "pointless" or "useless"... that's like, the opposite of "necessary". Just sticking with an English word like "foundation" would make sense here - necessary, but not something a customer or user typically sees. Even further, metaphorically, fixing bugs = stabilizing the foundation. Using foreign words just to sound fancy and ending up with the wrong meaning is just annoying.

jlawson 2y ago
Alternatives: foundation, plumbing, substructure, support work, support structure, backend, sustainability, backstage, logistics
- myhf 2y ago
  "necessary but not sufficient"
kitten_mittens_ 2y ago
A good chunk of the English lexicon was foreign at one point or another: https://en.wikipedia.org/wiki/Inkhorn_term https://en.wikipedia.org/wiki/Inkhorn_term
jessetemp 2y ago
They're saying the things that don't add value are muda (waste), not the foundational stuff. I only know that because my employer uses the same jargon. And since almost no one speaks phonetically-spelled-out Japanese, they always follow it with (waste) in parentheses
jdlshore 2y ago
“Muda” is a technical term with a specific meaning. (Google it.) It originated in the Toyota Production System, then migrated to software development via the Poppendiecks’ “Lean Software Development.” It doesn’t mean “pointless” any more than legacy software means “an inheritance.”

dboreham 2y ago

The executive summary of this article (for me) is: burst the delusion bubble around your stake holders. This is admirable, but that delusion existed in the first place for a reason. The reason still exists.

a_c 2y ago

I might have skimmed too fast. Productivity seemed not defined? IMO productivity is all about making product that get used. And profitability is how much people willing to pay to use such product.

neilv 2y ago

I wonder how well the engineering/product would've gone, in that case, without that productivity-measuring flavor of leadership being asserted.

w10-1 2y ago

Risk is lost in the discussion but very relevant to management. In theory, in the most productive engineering organization, each person is the expert at what they're doing and work is factored to avoid overlap. This happens somewhat naturally as people silo (to avoid conflict and preserve flexibility). This is actually the perfect system, if your people are perfect: their incentives are aligned, they're working with goodwill, etc. But that makes every person critical, few people can operate at that level all the time, and fewer still want to work alone doing the same thing ad infinitum. Also, needs change, re-factors work and its mapping to workers. So then you expand the scope to productivity over time with accommodation for change - i.e., capacity. But by definition, capacity far exceeds output, and is thus unobservable and unmeasurable. So instead you experiment and measure: switch roles, re-organize work, address new kinds of problems and prototype new kinds of solutions, bounded by reasonably anticipated product needs. And to inject this uncertainty, you actually have to inject a lot of countervailing positivity, to deflect the fear of measurement or failure, to encourage sharing, etc. Unfortunately, these experiments and freedom from consequences is pretty indistinguishable from the bureaucratic games and padding that productivity measures are designed to limit.

deleted 2y ago

[deleted]

OJFord 2y ago

With articles like this I always think it'd be really interesting to hear from others on the team. I find it really hard to imagine there's not going to be pressure (either self-inflicted or from above) to 'game' it, or how do you decide what's gaming and what's pragmatic anyway - perhaps better said I find it hard to imagine people don't feel like it's being gamed, that they're having to add value where they think they should be maintaining, even if OP or whoever deciding that thinks it's the right call. (And to be clear I can't say and am not saying which is right, from over here with zero information about any such decisions.) It could be fun to have a platform for honest blogs about work stuff where it's grouped by people verified to work (or have worked) at the same place. (Or, I realise as I write that, a descent into in-fighting and unprofessionalism..)

mdgrech23 2y ago

"but the ceo had a scheduling conflict so he couldn't come to this thing that was really important to him". Does that ever make sense to anyone? It never made sense to me. I think it's about classism. These guys only want to hang out w/ the lower class if they're at the front of the room talking otherwise they want to come off as mysterious and too busy to have any time. What a load of shit.

rhelz 2y ago
cf. with the observation that "leadership teams like accountability a lot more when its other people being held accountable."

booleandilemma 2y ago

There's no shortcut to paying attention. That is, paying attention to who's doing the real work. Any metric can be gamed.

ozim 2y ago

I would point out that Martin Fowler, Kent Beck and Gergely Orosz were writing that you cannot measure productivity of a single team member. *You can get a rough sense of a team's output by looking at how many features they deliver per iteration. It's a crude sense, but you can get a sense of whether a team's speeding up, or a rough sense if one team is more productive than another.* So in that sense "value add time" will be a valid metric. But it is still not a single number that one can give without context to the CEO, that CEO still has to ask questions like "we usually have X time spent on fixing bugs, aren't you gaming the system to get your bonus?". That is the CEO job to understand the metric not to be gamed...

deleted 2y ago

[deleted]

jlas 2y ago

Don't overlook sources of anti-productivity which might be easier to measure like the time it takes to build & test code changes.

danielovichdk 2y ago

Productivity is a totally irrelevant metric. It takes 2 people, 5 years to built a 50 foot sailboat. It takes 10 people, 4 months. Which is most productive? That depends on the price and the quality of course. Never measure for quantity. Always measure for quality in the environment you are competing in. And since most professional work is about making money/use value, it would be a could start to measure how much money/use value people are contributing to. If people are not actively contributing to this they are not productive.

siliconc0w 2y ago

I like DORA metrics as a guideline (Deployment frequency, lead time for changes, change failure rate, time to restore) which aren't perfect but are fairly straight forward to understand and measure. I might look at number of production touches(manual interventions in the delivery pipeline), story points estimated, story points delivered, number of pages, number of bugs created/closed/closed w/ change, number of errors/exceptions. It's possible to hide technical debt here but it eventually start to manifest in one of the stability metrics (bugs, exception, production touches, etc) or a drop in velocity . Though is more about engineering delivery of the product roadmap. I do like the 'bets' framework in the OP to figuring out whether the product roadmap actually results in meeting the desired goals for the organization (growth, profits, whatever) but these should be measured as different things.

rhelz 2y ago

I'm reminded of a conversation I had with a colleague from Georgia, who had a beautiful southern drawl. He said, "The way Ah measure Ahr Oh Ah is Ahr over Ah." Assigning productivity numbers to individual engineers is enormously difficult-- but at the level of VP of all programming, it seems simple: At the end of a quarter, you know how much you spent on programmers during that quarter. And a couple of quarters after that, you should know how much value was added. Why not calculate ROI as R over I???

WatchDog 2y ago

It seems like the main problem with any productivity metric, is that once you tell your workers what the metric is, it will be gamed. However if you collect the metrics in secret, and only use them as a guide for where productivity could be improved, it seems they could be more useful. I wonder if any engineering managers are doing something like that? I guess it would be hard to scale to a large organization, as the you would need to share the metrics with the line managers.

senkora 2y ago
I suspect that many quietly effective employees are doing something like that, and that it is a smart way to increase the performance of yourself and your team. I also suspect that it is not possible to share the metrics, and that you would need to do "parallel construction" to justify your decisions without revealing your real methodology.

Willish42 2y ago

> Actually, that’s the root of my org’s current capacity problems. They weren’t cheating a metric, but they were under pressure to deliver as much as possible. So they deferred a bunch of maintenance and took some questionable engineering shortcuts. Now they’re paying the price. > Unfortunately, you can get away with cheating this metric for a long time. Years, really. It’s not like you cut quality one month and then the truth comes out the next month. This is a metric that only works when people are scrupulously honest, including with themselves. I fear this is what's happening at basically every major tech firm in the world right now. We might not fully realize the stupid breakages and tech debt of these decisions (namely: layoffs and large refactors of the product and system to inject AI everywhere) for _quite_ some time. I fear it'll be just like "social first" and "mobile first" iterations of the web that create a bunch of cruft to be cleaned up later.

matchagaucho 2y ago

The best companies can churn their metrics quarterly. Early days of FB were often driven by MAU. Then P95 performance became a priority, leading to higher DAU.

cess11 2y ago

If a boss came to me and said 'hey from now on we're going to describe your work in terms of gambling' whatever loyalty I had would have collapsed.

aryehof 2y ago

It seems to me that “productivity” is being used here carelessly. It is a ratio of effort and resulting output. High productivity will produce more output given a specific effort, than low productivity, given that other factors like quality are constant. Frankly, in software we cant even determine the extent and relationships between requirements, effort, quality and output in any predictable way. Given this, isn’t an attempt to produce some resulting measure like productivity meaningless?

thefaux 2y ago

All of this feels like a side effect of a low trust environment. I respect the author's creativity, but the environment described sounds highly dysfunctional. In my opinion, if the ceo is demanding an arbitrary metric from his subordinate without being able to articulate what they are looking for sufficiently that the subordinate has to invent a made up number just to justify their existence, that sounds like poor leadership to me. Of course, this is only based on the author's side and perhaps I'd feel differently if I heard the other side.

bennyelv 2y ago
It is (source: I work in an org like this), but your aim as a leader, the VP in this case, is to do whatever is necessary to enable your team to succeed, given the constraints that exist. Ineffective company operation and other people’s hangups and shortcomings are constraints. Does the nirvana where you are never limited by the ineffective operation of others exist anywhere? Some companies are better, some are worse, but all of them are collections of people with limitations at the end of the day!

la64710 2y ago

TLDR This is an article about a new way to measure software engineering productivity. It discusses the challenges of measuring productivity in software engineering. The author proposes a metric called value-add capacity. This metric focuses on the percentage of time engineers spend on activities that directly add value to the customer. The author acknowledges that this metric can be easily cheated. However, he believes it can be useful if used honestly.

dusted 2y ago

I misread the title as the more interesting "Is productivity a useful measure?"

jerrygoyal 2y ago

how to measure dev productivity? you can measure it by how many times a dev rewrites the code before the code gets merged.