12 ms·

Engineering productivity can be measured, just not how you'd expect

170 points by tomasrb 6y ago

strangattractor 6y ago
The best teams appear to be doing nothing. If as a general rule things are always in a fire drill then your software sucks (there are other reasons for this). We always measure how many things are fixed. New features added. Rarely how much time is spent baby sitting machines etc.
- emteycz 6y ago
  Sadly business people want to fire you when you're seemingly doing nothing (as in coming to work on time, doing your job without stress and conflict, going home on time)... Even if all the business goals were met, they just can't stomach someone not working their asses off 24/7 for them.
  - ndiscussion 6y ago
    It's about power not money.
    - karmakaze 6y ago
      That's actually insightful. Wish I'd known this early in my career. Often, I'll be doing a great job with no complains from peers or project deliveries but seemingly 'tolerated' and not liked by any management types as they had no influence, I just did what needed doing of my own accord. Or maybe I didn't need to know, ignorance was bliss.
      - ndiscussion 6y ago
        Check out this post: https://daedtech.com/defining-the-corporate-hierarchy/ https://daedtech.com/defining-the-corporate-hierarchy/ I really recommend his book Developer Hegemony, although it was very depressing and I haven't quite recovered a year later :D
        karmakaze 6y ago
        Enough of that is funny 'cause it's true or sad but true. I work at a big co but not a full-on enterprise corp. I value my own time and work, more-or-less in alignment with broader initiatives. Learning new things and technical challenges makes the journey interesting, and it's fun/easy to get work done with most people. Sometimes I have varying degrees of communications (tech vs non-tech or grok/non-grok conceptual) issues but for the most part leadership has enough technical sense to make decent choices. So far, so good.
  - maximente 6y ago
    well, yeah. if you don't have air cover from a team or manager and don't appear to be engaged in shared suffering, it's no surprise people are going to ask questions, both from above and equal level. other labor will resent your non-suffering (how come /they/ get to leave on time?) and management will wonder whose reports get such a cushy setup and why.
    - emteycz 6y ago
      The thing is, there might be no suffering one is "avoiding" - the business people are often sitting around drinking coffee and beer, cracking jokes, watching developers' backs.
- snidane 6y ago
  The best way to motivate devs to build truly robust systems is to let them go home at 2pm when things are finished and production process is running smooth. However most organizations would pile more work onto devs if they finish early, so devs then compensate by taking more time on their current tasks. Why finish them early, when you'll be thrown another one right away.
  - kamilafsar 6y ago
    Finish what? Where do you draw the line of “work for today”? I don’t think motivated devs want to go home at 2pm. Speaking as a dev, the best way to motivate me is to give me some head cracking challenge, trust (no reporting) and autonomy (don’t tell me how to do my work).
    - BossingAround 6y ago
      > I don’t think motivated devs want to go home at 2pm. I can understand the attitude when one's in their twenties, has no significant other, kids, or other commitments. Me? I just get tired. I'm very motivated and like what I do, but at some point, it's just silly to stay behind the screen. And sometimes, that point is even earlier than 2pm. That's ok. We aren't machines to be working like a Swiss watch.
      - Jtsummers 6y ago
        I was most productive when my office let us take a few hours a week to exercise. I wouldn't use it if I had anything critical pending, but otherwise I would duck out an hour early several times a week to go get in a good long run (5-10km). It felt like they had more respect for my time and well-being. When they cut it (with bullshit reasons [0]) "productivity" did not improve in the office. Instead, projects expanded to fill the time. Those 3 hours or so we got to use at the gym before became 3 hours spent to produce the exact same total work. There was zero motivation to utilize it "properly" by using it to get ahead on anything. Every project continued to hit the same deadlines they were hitting before. [0] One reason I say it was BS, we were typically on or ahead of schedule on projects. Teams that were behind weren't making use of this time while they were behind, they weren't permitted to.
- mbrodersen 6y ago
  You nailed it. The best engineers I have ever worked with never worked much. The bad engineers were always super busy trying (and failing) to fix the problems they put in the software in the first place.
bjornsing 6y ago
> Engineers will become more focused and engaged, managers will become more effective and empathetic, and companies will build faster with higher quality. Engineering will rise to a whole new level. A bit too much hyperbole for my taste, given the less than groundbreaking ideas.
- horsawlarway 6y ago
  I mostly agree - This doesn't seem to be adding any real visibility that velocity tracking (a la - agile) wouldn't give you already (not that I'm advocating for agile, mind you...). Consider - I have two teams, with the same staffing levels and the same general seniority. For this example, lets assume each is a team of 5, with 1 tech lead, 2 seniors, and 2 juniors. Both teams have the same approximate meeting count, both work on the same stack with the same dev tools. Team A consistently releases new features faster than team B. Why? Because if the answer is "Find the blocker" aren't we right back at > "your engineering leaders will simply justify failures, telling stories like "The customer didn't give us the right requirements" or "We were surprised by unexpected vacations."" except with blockers this time? Maybe Team A is actually just better than Team B. Maybe Team B is actually working on a feature set that has more inherent complexity. Maybe Team A releases faster but also has more incidents in prod. Maybe Team B releases a larger changeset on average. None of this is getting addressed or answered. ---- None of that is to say that measuring blockers isn't a useful idea, but it's certainly not some silver bullet.
  - majormajor 6y ago
    Those blockers should be things that give you falsifiable stories, no? So if someone says "it's because we're blocked on [slow delivery of designs from another team]" and you measure that specifically, and then improve it, and you notice the team's output hasn't changed, you've learned something. I've certainly seen those reasons before, but haven't seen people turn them into specifically measured things versus "ok let's see if we can improve it" with often little or ineffective followup.
  - thinkersilver 6y ago
    It helps to instrument the journey of your work from jira all the way to build, deployment, run and monitoring (observation). From that you can get measurements on how long each stage takes and the duration of each transition. From there you can compare team A and B. The transition times is where the human time cost usually sits. Just getting the time when a Jira or feature is raised, to the time it is picked up to the time of the first commit to the time of the first test and final build does already give you valuable insight. The points you raised towards the end can be answered if observability of your CI/CD pipeline is actually in place or at least a place to start a line of inquiry. Naturally the blockers will be aggregated into some of the values but as you work through the journey, they will start clustering at certain stages and maybe highlight a significant problem that needs to be addressed. There's a wealth of data being left on the table that can help inform management decisions.
tyingq 6y ago
"Measuring blockers" may make unfair scapegoats out of undersized teams.
- andor 6y ago
  Can you explain why you think this will happen to undersized teams more often? If it's an indication that something is wrong with the team, I think that's still helpful. For instance, if the team does not have the right experience and are often blocked waiting for someone else's answer to a question, it might make sense to work on team composition. It's not about blaming people or teams, but about improving flow.
  - novok 6y ago
    Often you literally don't have hiring headcount allocated to your team. Sadly the solution is to make the team implode to force the company to fund it properly.
  - tyingq 6y ago
    Just have seen it quite a lot. QA, Ops, etc, are often viewed as a cost center and thus underfunded. It's common to then challenge the "productivity" versus looking at whether the team is undersized or funded.
    - musicale 6y ago
      > It's common to then challenge the "productivity" versus looking at whether the team is undersized or funded. This exactly. The typical management "solution" is to identify and fire "poor performers" (thereby creating fear, mistrust, and an incentive to look for other jobs) while ordering everyone else to "work harder" for longer hours (with the implied threat that you will lose your job if you don't.)
- majormajor 6y ago
  If your team is a constant blocker because you're undersized, you should be given more headcount, or some responsibilities should be shifted around. That's then necessary to upper management in order to remove the blocker. (Assuming an org that's making a genuine attempt to improve, versus just a cynical blame game. But in the latter case... you're screwed sooner or later anyway.)
snidane 6y ago
The article is correct to point out that in case of no process measurement, politics, not results, will start to dominate. Measuring process inputs will favor Sisyphean work. Appearing to be working, hours punched in and butts-in-seats will be more valued than results. Many companies are stuck in here. Measuring proxies of outputs such as lines of code, or number of tickets closed, as the article mentions, only leads to people gaming the system. Measuring the actual value delivered is obviously the best thing to do. However it is often difficult to even define value and the amount of it created. Which is always a problem especially for inhouse projects or in the absence of direct contact with the market. I like what the article suggests - measuring process flow instead of measuring inputs or outputs. Can't say exactly why I like it but it seems that engineer types are naturally motivated to perform and learn, so giving them enough space is a good way to get working systems as a side effect.
- mr-wendel 6y ago
  I like it too, but I think it misses something crucial: how often is the team unblocking others? This, naturally, is at odds with minimizing interruptions. On one extreme there is no shield around engineers and they experience constant whiplash with "code oracling", shifting priorities, obscure trivialities, and other things that can drive people insane and can prevent meaningful work from being completed. On the other hand, sticking perfectly to "we're committed to this sprint and unless its on fire you wait in line" and much of your business becomes unnecessarily rigid and painful. That, IMHO, is much worse. Finding a balance is crucial, and to me thats just as interesting of a datapoint.
  - visarga 6y ago
    > "code oracling" Congrats, according to Google it seems like you invented a new word. Could you define it for us?
    - mr-wendel 6y ago
      A little late on the response, but... People want to know a specific behavior of a system. How often does X happen? What kind of things can trigger Y? Maybe there is documentation, but even if there is nobody trusts it. If folks could just easily and see they would have. The only solution is to start reading the source code. Perhaps you even wrote it, once upon a time. Nonetheless, all you can do is scan the code, built a mental model, try to see if its testable outside of production, and procure an answer. In my experience legacy systems, especially when laden with tech debt, require an awful lot of this if you're doing anything more than just keeping the lights on.
dazzawazza 6y ago
The more time goes by, the more I value what I learned from Peopleware (DeMarco, Lister) 20 or so years ago. It seems so clear, precise and uncluttered compared to modern efforts.
- ndiscussion 6y ago
  I read Peopleware last year and totally agree, it's the masterclass. Along with Mythical Man-Month, it's clear that we didn't need any other management books.
- stuxnet79 6y ago
  Peopleware is a classic. It's genius lies in the fact that it's able to shine a light on the important aspects of productivity and work life that fall between the cracks in our quest to 'measure' productivity and be 'efficient' (Taylorism). The big issue it has, IMO, (which other DeMarco books like 'Slack' share) is that it does not provide actual case studies that would convince a typical executive / manager working in a high pressure / competitive environment to change their methods.
andrey_utkin 6y ago
Engineering activities are economic activities. Engineering activities must make sense as economic activities. Engineering activities productivity can be measured the same way the productivity can be measured for any economic activities. See "Internal Market Economics: Practical Resource-Governance Processes Based on Principles We All Believe In", Book by N. Dean Meyer. The posted article came so close to the idea of the right measurements, I was intrigued and was like "yes, say it!", but the author didn't say what I expected, instead they pointed at quantifying calendars and commit logs. What a bizzare self-contradiction.
- dmos62 6y ago
  So what is that true way to measure productivity you alude to? By the way, engineering can be an economic activity, but it doesn't have to be. Could be an art as well, for example.
  - andrey_utkin 6y ago
    Simply: does revenue exceed cost of running it? The book I referred to answers the "but engineering team doesn't have revenue". Engineering can be art, sure, and if you like, we can even discount the idea that art creation can be seen as economic activity. But is there a need to measure productivity then?
    - username90 6y ago
      > Simply: does revenue exceed cost of running it? Cost of running it includes engineering salaries, so isn't a good measure to include when you measure productivity of engineers. A productive engineer would be worth a higher salary, but if you include the cost of that higher salary reducing his value would that also reduce the salary you want to pay? Doesn't make sense.
      - dmos62 6y ago
        Also, what if engineering is good, but sales is bad? Engineers might exceed their productivity goals, but it wouldn't amount to profit.
p0nce 6y ago
Here is an idea: measure added value. Realize it's produced at the bottom and consumed at the top of the hierarchy. Remove yourself from the organization as a pure consumer of added value.
cccc4all 6y ago
The process of abstracting away people into units of productivity is common approach to Software Engineering management. It has failed every time. When the latest management fad fails to live up to expectations, new terminology are created to hide the failed management fad. New terminology simply replaces the old terminology and they can now sell new and improved management concepts again and again. Software Engineering is extremely difficult. That's why successful Software Engineering companies are highly valued as unicorns. The companies have temporarily captured the right type of people tackling right type of problems and delivered right type of Software Engineering solutions. The process can not be replicated. That's why it's so valuable and worth Trillions of dollars. There's no other Microsoft. There's no other Apple. There's no other Google. These are unique companies with unique products that provide value to customers. It took years and decades of Software Engineering man hours to iterate until they delivered the valuable software solutions. Lot of these Software Engineering management fads are trying to capture something that does not exist. Productive Software Engineers are exceptional. Sooner Software Engineers understand that paradigm, the better they will be able to value themselves and be more effective.
- groby_b 6y ago
  Productive software engineers aren't "exceptional". If you focus on removing obstacles, you can make most software engineers productive. The point is learning from those who've succeeded before you, instead of claiming it's just magical chemistry happening. Or worse, starting again from first principles. No, that won't produce product-market fit, nor does it open up once in a lifetime opportunities, but it allows teams to execute well. The belief that productive software engineers are unicorns, however, and need to be valued above all else? That's what destroys companies.
  - cccc4all 6y ago
    There are reasons why terms like 10x engineer, Rockstar engineer, Ninja engineers are thrown about in the industry. Google paid Anthony Levandowski $120 Million to lead their autonomous car project, before he went to Uber and the legal troubles. Uber was paying him more than Google. Facebook paid $16 Billion for What's App, which only had around 50 engineers at the time. Facebook was buying the IP and the productive software engineering members of the team. There are countless other examples. No amount of management process can create productive software engineers. That's why companies recruit productive software engineers from other companies. That's why companies buy other companies that has productive software engineers with proven products. Productive Software Engineers are exceptional. Their work are worth Trillions of dollars and generates Billions of dollars in revenue every year.
necco908 6y ago
Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure with the intent to improve how your team works. If a manager can measure at the team level and open up visibility into the development process, they can hopefully find where things get frustrating (ex. waiting for someone to review a PR). That said, there seem to be more mature tools out there than OKAY's beta. There's a discord server called dev interrupted that talks about this stuff a lot.
- sdesol 6y ago
  I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rvfx https://imgur.com/DK7rvfx (gitlab) You'll find they both have a large cluster of developers in zone 2. For developers to exist in zone 2, they have to have medium to high impact on the code that they worked on, but not clash with others. If you look at the vuejs-next repository https://imgur.com/eDAOyPW https://imgur.com/eDAOyPW (vuejs) You can see it's actually a pretty fragile project, since Evan is responsible for pretty much everything. Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. Edit: The metrics that I'm showing is limited to the last 90 days for Typescript, Javascript and CSS code.
  - majormajor 6y ago
    > Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. This is a dangerous conclusion, especially for a business. If you're in pure maintenance mode, maybe... but otherwise... You want people who can pitch in anywhere, who can fix things rapidly, who can build new solutions quickly when required, and who know your business inside and out. You just don't want knowledge siloed there, so you want to make sure other people are also on the path to being expert on the various areas.
robotfelix 6y ago
A lot of the questions listed heavily overlap with what I would expect to be covered in team retrospectives. A team reflecting on progress at regular intervals will naturally bring up processes, meetings, tools, etc, and a manager or team lead can easily add these questions into the mix for reflection and/or discussion as part of this process. The key distinction from this seems to be hidden in the line "Finally, turn all these questions into metrics" - but the article could definitely do a better job of highlighting the differences between their "solution" and retrospectives (as well as more general good management practices like talking to your team). There are some interesting ideas in there that seem tied to their product (https://www.okayhq.com https://www.okayhq.com), but the article doesn't really progress far enough past the high level ideas to be practical and rather a lot is left as an exercise for the reader!
mlthoughts2018 6y ago
But how do you get anyone up the food chain to care? I’ve never witnessed that anywhere I’ve worked. If you took things like team surveys, code review load, meeting load, etc., and tried to make a case that changes are needed, you’ll get laughed out of the room. The only metrics that will matter are “did you get the stuff done that we wanted?” (where “stuff we wanted” is defined vaguely and the meaning shifts to suit product leaders ever changing political landscape), “did you get it done really fast?” (where “fast” is vaguely defined according to product political circumstances and never puts any weight on engineering estimates, staffing needs, or resource limits) and “did you get it done cheaply?” (where “cheap” is defined vaguely based on various internal politics and budget turf wars as well as larger company financials - and when the money is good and no one looks too hard at this, it allows sweeping other issues under the rug, no one cares if you burnt a quarter working on the wrong problems). Assessing effectiveness, in principle, is purely a political concept that operates from the top down. This has been true in every company I’ve worked for or knew a colleague or friend who worked there - from tiny “lean” startups to extreme cultures like Bridgewater to every mid-sized or large tech company.
dacracot 6y ago
> Just as a sports team wins or loses together, so too should the engineering team be treated as the fundamental unit of success. A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? Too many times I have see engineering teams as only a team on the org chart In reality they solve tickets as individuals with only a small interaction from pull requests. Otherwise they might as well not even know each other. They are a team not as in basketball or football, but like golf where once you get to the tee, it's you and only you to get the ball in the hole.
- babesh 6y ago
  Baseball
- cambalache 6y ago
  Not only that but as every sports fan know the compensation in sports team vary tremendously, even for similar "roles". People in tech would be shocked (with reason) if that were the case in the IT world.
- voodootrucker 6y ago
  > A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? This is why I like XP. Their teams really are teams like you say. Though I think in many dev shops you can be team like. Someone might like refactoring and cleanup. Someone else is good at rapid prototyping. Another architecture. Sometimes a great dev is just the one who can take the unglamorous tickets and get them done at a sustainable pace. Or someone who is good at devops, teaching, or morale building. Sometimes just communication. Everyone has different strengths. No one would ever say "who's the best (American) football player?" because you'd have to ask "who's the best kicker, tight-end, defensive lineman". They are all different roles. To think that football would have the level of awareness that it cannot be measured as a single dimension makes it sad and laughable that people reduce programming skill down to one most of the time.
  - lazyasciiart 6y ago
rajacombinator 6y ago
Let’s see an article about measuring manager productivity.
- wuweipractioner 6y ago
  IMO a manager's productivity is measured based on how well their team performs(or how well they present their team's productivity) and this article provides an approach on how to do so.
  - prepend 6y ago
    So the more blockers a manager removes, the better the look to their manager. It’s turtles all the way up. I actually think that one’s compensation is correlated to how ambiguous the job duties are and how hard it is to measure success, given employment. Stuff that’s easy to measure gets commoditized. Stuff that’s hard to measure, but still important is hard to staff, creates risk and is easy to mitigate with money.
mbrodersen 6y ago
The best engineers don’t do much work. They are excellent at delivering the software needed to solve a problem fast and without problems. The worst engineers are always super busy. Fighting fires, starting new fires, rewriting code that doesn’t require rewriting, introducing new shiny tech that will break things in new busy work creating ways etc. etc.
- purplecats 6y ago
  the latter get promoted and favored by management. go figure.
obiefernandez 6y ago
> If your team is full of competent, driven engineers, removing their blockers is the fastest way to enable forward movement. That's a HUGE if right there. In my experience, the most problematic teams I've ever worked with were simply full of incompetent and/or unmotivated engineers, and for whatever reason it was impossible to replace them.
- prepend 6y ago
  I think the challenge with this is that blockers only get removed once and super awesome teams preemptively remove blockers before they block. So while it’s good to remove obstacles and maybe helpful to measure if obstacles exist that’s bad, it doesn’t mean much when no obstacles exist and none have been removed in a long time. Maybe they are super awesome and chugging along from all those removed obstacles. Maybe they are super sucky because they don’t see things as obstacles.
sinuhe69 6y ago
If you only measure blockers as an index for productivity then two teams with the same number of blockers should always have the same productivity, regardless how the people are? I cannot fathom how wrong this “measurement” can be. Yes, reducing blockers is important but that is not be the only thing the world cares about.
didibus 6y ago
> Each engineering team is unique, so its blockers will be specific. It's not so simple as "more maker time is better." If your engineering team is new or temporarily misaligned on key goals, more meeting time might be the answer. What never changes is the need for measurement and well-considered, deliberate decisions And we came full circle back to measuring a teams productivity being an art form. Maybe you have a metric now: Hours spent in meetings per week, but nobody knows how much is too much and how little is too little. How do you measure the impact of more meeting on productivity? Or the impact of less meeting? If your measure of productivity is "time in meeting"? This is a circular dependency.
- ec109685 6y ago
  That’s why they suggest surveys to gauge and looking at how much “maker” time engineers have.
didibus 6y ago
One thing that wasn't covered here is that productivity is about: > the effectiveness of productive effort and > the state or quality of producing something Those are quoted from the dictionary definition of productivity, and that definition in my opinion outlines a great insight. Productivity is about the "product" first and foremost. One thing that's often missing is teams don't have ways to quantify how good the product itself is. Most teams will instead pivot to trying to measure their rate of change to the product. This doesn't mean you have become more productive, because software product is not like food production, more of it is not always better. Better software will instead be about it being more ingenious, more intuitive, more tailored to the problems of its users, more responsive, with fewer malfunctions, etc. So to me, this whole thing of trying to measure "productivity" which disregards the product from the equation is incomplete. It's trying to measure developer efficiency at changing things, without care if the changes are for better or worse. This includes things like measuring velocity, or line of code, or ticket closed. But it also includes what this article proposes, number of meetings, time to complete code reviews, developer satisfaction with their tools, etc. All these are trying to see how quickly can developer make changes without ever measuring if the change produces a better product, thus if it actually made the team more productive at effectively positively improving the product. I'd like to hear about ways to measure software product state and quality. If we had those, you'd have an easy way to know how productive a team would be by seeing how quickly they can improve the product.
- kqr 6y ago
  Yes! There's a strong tendency toward measuring unvalidated proxies because they are easy or convenient to measure, without even constructing a hypothesis about how these proxies relate to more important things, much less testing it. Measuring productivity has to start with what matters: do your end-users (these are not necessarily your customers) like what you do? And often it is hard to get this information. Just asking them will lead to biased responses, but there are ways to deal with that. I happen to be in a business (let's call it food) where I can observe the end user using more of their money with our customers rather than their competitors when we do things right. That's a strong signal -- probably stronger than asking them -- so a very fortunate starting point. (Of course, there are still confounding terms like seasonality, economy etc to grapple with, but there are ways to deal with that too.) Starting with that one measurement that matters, one can begin exploring proxies. Set up a hypothesis: "Velocity would be easier to measure and the number would be available faster. Does it correlate with our one good metric?" And then you run the experiment. It might take weeks or months to get back the end-user-happiness data that corresponds to this week's velocity, so these tests are expensive (but pay off many times over when you find good proxies. In the end, you should be able to construct a somewhat sensible model of user happiness, and answer questions like, "if we hire another team member and therefore increase velocity by 3 %, how much happier will our users be? And what is that worth in sales?" When you can convert everything to the same unit of measurement (dollars are an easily explained option, but log-dollars are a personal favourite of mine) you get intense clarity and alignment around priorities and decisions. ---- All of that said, development speed, as defined in the Accelerate study in particular, is one of those generally good things you pretty much unconditionally want. The reason is given in the study and expanded on further in Reinertsen's Principles of Product Development. The reason speed is important is that successful product development is controlled by surprises. You will discover something tomorrow that will make you wish you had prioritised differently today, and being able to pivot quickly on surprises is how you both de-fang the biggest risks, but also how you throw yourself at opportunities before your competition even realises there is one.
caseyross 6y ago
One thing I really like about this approach is that it creates strong incentives for engineers to write maintainable code. Code that is expedient but brittle or hard to understand will be noted as blocking progress for future changes, while robust code can be promoted as blocker-free.
vasco 6y ago
The idea of identifying and removing a bottleneck isn't new. I don't know about randomly removing "blockers", it's generally widely accepted that only the bottleneck makes sense improving, everything else is wasted work as you'll just accumulate work in progress. People sometimes rebel a lot against this because "my code is a creative process and that's for factory lines" but I've never found myself worse off when improving a bottleneck, though you do have to fight the "but everything counts" crowd.
GizmoSwan 6y ago
Good work. That is how it should be measured. In economics as far as I know, they measure productivity via salaries. The more money people make, the more productive they are assumed to be!
- GizmoSwan 6y ago
  'Evaluations of an economy's productivity performance are made using a measure of real gross domestic product (GDP), which represents the constant dollar income (labour income plus profits) that an economy generates through domestic production, with the volume or constant dollar indices being calculated from the prices' 'DP=(GDP/Hours)*(Hours), (1) where Hours is the total number of worker-hours. Chart 1 depicts changes in each of these components over time. For the entire 1961-to-2012 period, labour productivity advanced at a 1.9% annual average, accounting for slightly more than half of the increase in GDP growth. The rest is attributed to hours, which increased at 1.5% per year on average. Aggregate GDP measures the returns to both labour and capital. Distributional concerns lead to questions about whether the share going to labour increases over time and, in particular, how productivity growth is related to real income.' https://www150.statcan.gc.ca/n1/pub/15-206-x/15-206-x2014038-eng.htm https://www150.statcan.gc.ca/n1/pub/15-206-x/15-206-x2014038...
nitwit005 6y ago
I have to disagree with the idea that sales is somehow becoming a scientific process due to CRM. There are openly gut feeling fields like how a call seemed to go, how "hot" an opportunity is, etc. Most of the ideas predate software as well. People had remarkably complex paper based sales systems.
- prepend 6y ago
  I’m not in sales but I remember my firm having sales pipeline meetings 20 years ago where sales people would talk about stuff that happened. It was all manual and anecdotal. So the sales pipeline was only in someone’s head or maybe a spreadsheet they hacked out once a quarter or whatever. I think the difference today that the author referenced is that with Salesforce stuff like number of leads, contacts, conversion, revenue and profit per conversion, etc is all captured and reportable in sort of real-time. There’s still lots of gut inputs like how the meeting went or having people estimate probability of closure, etc. But now it’s more systematic. I don’t know if it’s more successful, but it seems like I get more sales calls and repeat contact attempts.
marcus_holmes 6y ago
I don't actually see a productivity measure in all this, though. If there are more blockers, is the team doing better (because they're covering ground faster and finding new blockers faster). Or worse (because they're being blocked more)? If I have to report to the CEO on the Dev team productivity, do I tell them how many blockers we removed?
simon_000666 6y ago
or... Just use the metrics outlined in the accelerate book, which have been fairly well studied and have quite a lot of quantifiable evidence to back them up. E.g. Deployment frequency Lead time for changes MTTR Change failure rate A quick google reveals a fair amount of existing material : https://leadingagileteams.com/2020/04/07/forget-dumb-productivity-measures/ https://leadingagileteams.com/2020/04/07/forget-dumb-product...
wiz21c 6y ago
FTA > Productivity in engineering therefore naturally increases when you remove the blockers getting in the way of your team. The guy just discovered the job of the PM or what ?
NumberCruncher 6y ago
I am surprised what a discussion an article can kick of which sole purpose is generating views for people.ai...
NotPavlovsDog 6y ago
> Instead of measuring some approximation of engineering output, software teams should measure actual, observable metrics that directly correlate to effectiveness. I prefer to measure value created and partially tie it to compensation, directly and indirectly. For a dev team for a trading platform, we had an index of income created by product, approved by team and stakeholders, that affected total compensation paid to team. This is a specific example. The general principle is let tech teams make decisions and compensate them for value created, at least partially. Otherwise, when you don't want to share your money, "measurement of productivity / effectiveness" comes in. Because when you measure money, people ask, "why am I not getting more when I make you more"? But if you're not measuring money, why do a business?
- TheOtherHobbes 6y ago
  Because measuring money is hard and it's not easy to apportion credit. Person A has an interesting-but-not-exceptional idea and gets Person B to fund it. People C, D, and E code it. For most interesting-but-not-exceptional ideas you could replace all of these people with others. If you actually ran this an experiment in a Monte Carlo-but-real kind of a way, you'd find some collaborations would do well, some would do badly, some would fail completely, a few might explode (in a good way). How do you quantify the value of the relative contributions?
  - NotPavlovsDog 6y ago
    We've run a variety of experiments that worked for us. What is important is implementing the question in a way that works for the organization. How shall we quantify value, and how much of that index will affect compensation. It's answering the question "why do some people get paid more" in a transparent way.
- michaelt 6y ago
  > I prefer to measure value created and partially tie it to compensation, directly and indirectly. How does this apply to team that maintains your source control and continuous integration systems?
prepend 6y ago
I like that the article kind of gives up metricing and just says each team should figure it out. This is kind of similar to the no metrics he proposes. If I had to choose, I’d choose no metric over lines of code or number of tickets, etc. I think the issue is that when orgs get big enough they have lots of teams and having some comparability is useful to find lessons learned to spread among teams, etc. I don’t know of any metric that is truly objective for figuring out high productivity teams or individuals just by using it. I think there are some “vital signs” that you want projects to have, but don’t want to fixate on the actual value. Like you don’t care if someone’s pulse is 70 or 80 or 90, but you want to make sure pulse is checked. I think having reviews in git is helpful and not having any is something to look into. I think having contributions from other teams is a good sign, although absence isn’t necessarily bad. I think the positive is by showing how others are finding and asking questions or reviewing or contributing material so that’s probably reuse. I think encouraging information sharing through lunch and learn presentations is good, but is delicate to avoid gaming from people just “making the circuit.” Having an automated CI/CD is a good sign and if code is making it to prod without one, it requires looking into. Theoretically a healthy team will have all these signs, but you could have an awesome team with low numbers and a terrible team with high numbers. So these metrics wouldn’t be useful for detecting productivity and comparing across teams, but would be good for just finding big, lurking problems. I comically work in an org where there are whole teams not using source control, so the “only I can measure myself, give me more money” is a very real challenge.
fukmbas 6y ago
It's such a flawed assumption that engineers must "produce something". Engineers are not hourly labor. Experts must be retained simply for reference.

170 points by tomasrb 6y ago

The best teams appear to be doing nothing. If as a general rule things are always in a fire drill then your software sucks (there are other reasons for this). We always measure how many things are fixed. New features added. Rarely how much time is spent baby sitting machines etc.

emteycz 6y ago
Sadly business people want to fire you when you're seemingly doing nothing (as in coming to work on time, doing your job without stress and conflict, going home on time)... Even if all the business goals were met, they just can't stomach someone not working their asses off 24/7 for them.
- ndiscussion 6y ago
  It's about power not money.
  - karmakaze 6y ago
    That's actually insightful. Wish I'd known this early in my career. Often, I'll be doing a great job with no complains from peers or project deliveries but seemingly 'tolerated' and not liked by any management types as they had no influence, I just did what needed doing of my own accord. Or maybe I didn't need to know, ignorance was bliss.
    - ndiscussion 6y ago
      Check out this post: https://daedtech.com/defining-the-corporate-hierarchy/ https://daedtech.com/defining-the-corporate-hierarchy/ I really recommend his book Developer Hegemony, although it was very depressing and I haven't quite recovered a year later :D
      - karmakaze 6y ago
        Enough of that is funny 'cause it's true or sad but true. I work at a big co but not a full-on enterprise corp. I value my own time and work, more-or-less in alignment with broader initiatives. Learning new things and technical challenges makes the journey interesting, and it's fun/easy to get work done with most people. Sometimes I have varying degrees of communications (tech vs non-tech or grok/non-grok conceptual) issues but for the most part leadership has enough technical sense to make decent choices. So far, so good.
- maximente 6y ago
  well, yeah. if you don't have air cover from a team or manager and don't appear to be engaged in shared suffering, it's no surprise people are going to ask questions, both from above and equal level. other labor will resent your non-suffering (how come /they/ get to leave on time?) and management will wonder whose reports get such a cushy setup and why.
  - emteycz 6y ago
    The thing is, there might be no suffering one is "avoiding" - the business people are often sitting around drinking coffee and beer, cracking jokes, watching developers' backs.
snidane 6y ago
The best way to motivate devs to build truly robust systems is to let them go home at 2pm when things are finished and production process is running smooth. However most organizations would pile more work onto devs if they finish early, so devs then compensate by taking more time on their current tasks. Why finish them early, when you'll be thrown another one right away.
- kamilafsar 6y ago
  Finish what? Where do you draw the line of “work for today”? I don’t think motivated devs want to go home at 2pm. Speaking as a dev, the best way to motivate me is to give me some head cracking challenge, trust (no reporting) and autonomy (don’t tell me how to do my work).
  - BossingAround 6y ago
    > I don’t think motivated devs want to go home at 2pm. I can understand the attitude when one's in their twenties, has no significant other, kids, or other commitments. Me? I just get tired. I'm very motivated and like what I do, but at some point, it's just silly to stay behind the screen. And sometimes, that point is even earlier than 2pm. That's ok. We aren't machines to be working like a Swiss watch.
    - Jtsummers 6y ago
      I was most productive when my office let us take a few hours a week to exercise. I wouldn't use it if I had anything critical pending, but otherwise I would duck out an hour early several times a week to go get in a good long run (5-10km). It felt like they had more respect for my time and well-being. When they cut it (with bullshit reasons [0]) "productivity" did not improve in the office. Instead, projects expanded to fill the time. Those 3 hours or so we got to use at the gym before became 3 hours spent to produce the exact same total work. There was zero motivation to utilize it "properly" by using it to get ahead on anything. Every project continued to hit the same deadlines they were hitting before. [0] One reason I say it was BS, we were typically on or ahead of schedule on projects. Teams that were behind weren't making use of this time while they were behind, they weren't permitted to.
mbrodersen 6y ago
You nailed it. The best engineers I have ever worked with never worked much. The bad engineers were always super busy trying (and failing) to fix the problems they put in the software in the first place.

bjornsing 6y ago

> Engineers will become more focused and engaged, managers will become more effective and empathetic, and companies will build faster with higher quality. Engineering will rise to a whole new level. A bit too much hyperbole for my taste, given the less than groundbreaking ideas.

horsawlarway 6y ago
I mostly agree - This doesn't seem to be adding any real visibility that velocity tracking (a la - agile) wouldn't give you already (not that I'm advocating for agile, mind you...). Consider - I have two teams, with the same staffing levels and the same general seniority. For this example, lets assume each is a team of 5, with 1 tech lead, 2 seniors, and 2 juniors. Both teams have the same approximate meeting count, both work on the same stack with the same dev tools. Team A consistently releases new features faster than team B. Why? Because if the answer is "Find the blocker" aren't we right back at > "your engineering leaders will simply justify failures, telling stories like "The customer didn't give us the right requirements" or "We were surprised by unexpected vacations."" except with blockers this time? Maybe Team A is actually just better than Team B. Maybe Team B is actually working on a feature set that has more inherent complexity. Maybe Team A releases faster but also has more incidents in prod. Maybe Team B releases a larger changeset on average. None of this is getting addressed or answered. ---- None of that is to say that measuring blockers isn't a useful idea, but it's certainly not some silver bullet.
- majormajor 6y ago
  Those blockers should be things that give you falsifiable stories, no? So if someone says "it's because we're blocked on [slow delivery of designs from another team]" and you measure that specifically, and then improve it, and you notice the team's output hasn't changed, you've learned something. I've certainly seen those reasons before, but haven't seen people turn them into specifically measured things versus "ok let's see if we can improve it" with often little or ineffective followup.
- thinkersilver 6y ago
  It helps to instrument the journey of your work from jira all the way to build, deployment, run and monitoring (observation). From that you can get measurements on how long each stage takes and the duration of each transition. From there you can compare team A and B. The transition times is where the human time cost usually sits. Just getting the time when a Jira or feature is raised, to the time it is picked up to the time of the first commit to the time of the first test and final build does already give you valuable insight. The points you raised towards the end can be answered if observability of your CI/CD pipeline is actually in place or at least a place to start a line of inquiry. Naturally the blockers will be aggregated into some of the values but as you work through the journey, they will start clustering at certain stages and maybe highlight a significant problem that needs to be addressed. There's a wealth of data being left on the table that can help inform management decisions.

tyingq 6y ago

"Measuring blockers" may make unfair scapegoats out of undersized teams.

andor 6y ago
Can you explain why you think this will happen to undersized teams more often? If it's an indication that something is wrong with the team, I think that's still helpful. For instance, if the team does not have the right experience and are often blocked waiting for someone else's answer to a question, it might make sense to work on team composition. It's not about blaming people or teams, but about improving flow.
- novok 6y ago
  Often you literally don't have hiring headcount allocated to your team. Sadly the solution is to make the team implode to force the company to fund it properly.
- tyingq 6y ago
  Just have seen it quite a lot. QA, Ops, etc, are often viewed as a cost center and thus underfunded. It's common to then challenge the "productivity" versus looking at whether the team is undersized or funded.
  - musicale 6y ago
    > It's common to then challenge the "productivity" versus looking at whether the team is undersized or funded. This exactly. The typical management "solution" is to identify and fire "poor performers" (thereby creating fear, mistrust, and an incentive to look for other jobs) while ordering everyone else to "work harder" for longer hours (with the implied threat that you will lose your job if you don't.)
majormajor 6y ago
If your team is a constant blocker because you're undersized, you should be given more headcount, or some responsibilities should be shifted around. That's then necessary to upper management in order to remove the blocker. (Assuming an org that's making a genuine attempt to improve, versus just a cynical blame game. But in the latter case... you're screwed sooner or later anyway.)

snidane 6y ago

The article is correct to point out that in case of no process measurement, politics, not results, will start to dominate. Measuring process inputs will favor Sisyphean work. Appearing to be working, hours punched in and butts-in-seats will be more valued than results. Many companies are stuck in here. Measuring proxies of outputs such as lines of code, or number of tickets closed, as the article mentions, only leads to people gaming the system. Measuring the actual value delivered is obviously the best thing to do. However it is often difficult to even define value and the amount of it created. Which is always a problem especially for inhouse projects or in the absence of direct contact with the market. I like what the article suggests - measuring process flow instead of measuring inputs or outputs. Can't say exactly why I like it but it seems that engineer types are naturally motivated to perform and learn, so giving them enough space is a good way to get working systems as a side effect.

mr-wendel 6y ago
I like it too, but I think it misses something crucial: how often is the team unblocking others? This, naturally, is at odds with minimizing interruptions. On one extreme there is no shield around engineers and they experience constant whiplash with "code oracling", shifting priorities, obscure trivialities, and other things that can drive people insane and can prevent meaningful work from being completed. On the other hand, sticking perfectly to "we're committed to this sprint and unless its on fire you wait in line" and much of your business becomes unnecessarily rigid and painful. That, IMHO, is much worse. Finding a balance is crucial, and to me thats just as interesting of a datapoint.
- visarga 6y ago
  > "code oracling" Congrats, according to Google it seems like you invented a new word. Could you define it for us?
  - mr-wendel 6y ago
    A little late on the response, but... People want to know a specific behavior of a system. How often does X happen? What kind of things can trigger Y? Maybe there is documentation, but even if there is nobody trusts it. If folks could just easily and see they would have. The only solution is to start reading the source code. Perhaps you even wrote it, once upon a time. Nonetheless, all you can do is scan the code, built a mental model, try to see if its testable outside of production, and procure an answer. In my experience legacy systems, especially when laden with tech debt, require an awful lot of this if you're doing anything more than just keeping the lights on.

dazzawazza 6y ago

The more time goes by, the more I value what I learned from Peopleware (DeMarco, Lister) 20 or so years ago. It seems so clear, precise and uncluttered compared to modern efforts.

ndiscussion 6y ago
I read Peopleware last year and totally agree, it's the masterclass. Along with Mythical Man-Month, it's clear that we didn't need any other management books.
stuxnet79 6y ago
Peopleware is a classic. It's genius lies in the fact that it's able to shine a light on the important aspects of productivity and work life that fall between the cracks in our quest to 'measure' productivity and be 'efficient' (Taylorism). The big issue it has, IMO, (which other DeMarco books like 'Slack' share) is that it does not provide actual case studies that would convince a typical executive / manager working in a high pressure / competitive environment to change their methods.

andrey_utkin 6y ago

Engineering activities are economic activities. Engineering activities must make sense as economic activities. Engineering activities productivity can be measured the same way the productivity can be measured for any economic activities. See "Internal Market Economics: Practical Resource-Governance Processes Based on Principles We All Believe In", Book by N. Dean Meyer. The posted article came so close to the idea of the right measurements, I was intrigued and was like "yes, say it!", but the author didn't say what I expected, instead they pointed at quantifying calendars and commit logs. What a bizzare self-contradiction.

dmos62 6y ago
So what is that true way to measure productivity you alude to? By the way, engineering can be an economic activity, but it doesn't have to be. Could be an art as well, for example.
- andrey_utkin 6y ago
  Simply: does revenue exceed cost of running it? The book I referred to answers the "but engineering team doesn't have revenue". Engineering can be art, sure, and if you like, we can even discount the idea that art creation can be seen as economic activity. But is there a need to measure productivity then?
  - username90 6y ago
    > Simply: does revenue exceed cost of running it? Cost of running it includes engineering salaries, so isn't a good measure to include when you measure productivity of engineers. A productive engineer would be worth a higher salary, but if you include the cost of that higher salary reducing his value would that also reduce the salary you want to pay? Doesn't make sense.
    - dmos62 6y ago
      Also, what if engineering is good, but sales is bad? Engineers might exceed their productivity goals, but it wouldn't amount to profit.

p0nce 6y ago

Here is an idea: measure added value. Realize it's produced at the bottom and consumed at the top of the hierarchy. Remove yourself from the organization as a pure consumer of added value.

cccc4all 6y ago

The process of abstracting away people into units of productivity is common approach to Software Engineering management. It has failed every time. When the latest management fad fails to live up to expectations, new terminology are created to hide the failed management fad. New terminology simply replaces the old terminology and they can now sell new and improved management concepts again and again. Software Engineering is extremely difficult. That's why successful Software Engineering companies are highly valued as unicorns. The companies have temporarily captured the right type of people tackling right type of problems and delivered right type of Software Engineering solutions. The process can not be replicated. That's why it's so valuable and worth Trillions of dollars. There's no other Microsoft. There's no other Apple. There's no other Google. These are unique companies with unique products that provide value to customers. It took years and decades of Software Engineering man hours to iterate until they delivered the valuable software solutions. Lot of these Software Engineering management fads are trying to capture something that does not exist. Productive Software Engineers are exceptional. Sooner Software Engineers understand that paradigm, the better they will be able to value themselves and be more effective.

groby_b 6y ago
Productive software engineers aren't "exceptional". If you focus on removing obstacles, you can make most software engineers productive. The point is learning from those who've succeeded before you, instead of claiming it's just magical chemistry happening. Or worse, starting again from first principles. No, that won't produce product-market fit, nor does it open up once in a lifetime opportunities, but it allows teams to execute well. The belief that productive software engineers are unicorns, however, and need to be valued above all else? That's what destroys companies.
- cccc4all 6y ago
  There are reasons why terms like 10x engineer, Rockstar engineer, Ninja engineers are thrown about in the industry. Google paid Anthony Levandowski $120 Million to lead their autonomous car project, before he went to Uber and the legal troubles. Uber was paying him more than Google. Facebook paid $16 Billion for What's App, which only had around 50 engineers at the time. Facebook was buying the IP and the productive software engineering members of the team. There are countless other examples. No amount of management process can create productive software engineers. That's why companies recruit productive software engineers from other companies. That's why companies buy other companies that has productive software engineers with proven products. Productive Software Engineers are exceptional. Their work are worth Trillions of dollars and generates Billions of dollars in revenue every year.

necco908 6y ago

Measuring engineering can be a slippery slope towards stack ranking performance -- which ultimately hurts performance and culture. I think you have to measure with the intent to improve how your team works. If a manager can measure at the team level and open up visibility into the development process, they can hopefully find where things get frustrating (ex. waiting for someone to review a PR). That said, there seem to be more mature tools out there than OKAY's beta. There's a discord server called dev interrupted that talks about this stuff a lot.

sdesol 6y ago
I'm currently working on a solution that will try to quantify software development health, and what I've learned from analyzing thousands of popular open source projects, is you don't want a single individual to stand out. You want work to be fairly distributed to reduce knowledge risk. If you look at the busfactor section for the vscode and gitlab repository https://imgur.com/NfgvvTy https://imgur.com/NfgvvTy (vscode) https://imgur.com/DK7rvfx https://imgur.com/DK7rvfx (gitlab) You'll find they both have a large cluster of developers in zone 2. For developers to exist in zone 2, they have to have medium to high impact on the code that they worked on, but not clash with others. If you look at the vuejs-next repository https://imgur.com/eDAOyPW https://imgur.com/eDAOyPW (vuejs) You can see it's actually a pretty fragile project, since Evan is responsible for pretty much everything. Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. Edit: The metrics that I'm showing is limited to the last 90 days for Typescript, Javascript and CSS code.
- majormajor 6y ago
  > Based on what I've observed by studying successful open source projects, you actually want to discourage "very high impact" employees, since they introduce knowledge risk. This is a dangerous conclusion, especially for a business. If you're in pure maintenance mode, maybe... but otherwise... You want people who can pitch in anywhere, who can fix things rapidly, who can build new solutions quickly when required, and who know your business inside and out. You just don't want knowledge siloed there, so you want to make sure other people are also on the path to being expert on the various areas.

robotfelix 6y ago

A lot of the questions listed heavily overlap with what I would expect to be covered in team retrospectives. A team reflecting on progress at regular intervals will naturally bring up processes, meetings, tools, etc, and a manager or team lead can easily add these questions into the mix for reflection and/or discussion as part of this process. The key distinction from this seems to be hidden in the line "Finally, turn all these questions into metrics" - but the article could definitely do a better job of highlighting the differences between their "solution" and retrospectives (as well as more general good management practices like talking to your team). There are some interesting ideas in there that seem tied to their product (https://www.okayhq.com https://www.okayhq.com), but the article doesn't really progress far enough past the high level ideas to be practical and rather a lot is left as an exercise for the reader!

mlthoughts2018 6y ago

But how do you get anyone up the food chain to care? I’ve never witnessed that anywhere I’ve worked. If you took things like team surveys, code review load, meeting load, etc., and tried to make a case that changes are needed, you’ll get laughed out of the room. The only metrics that will matter are “did you get the stuff done that we wanted?” (where “stuff we wanted” is defined vaguely and the meaning shifts to suit product leaders ever changing political landscape), “did you get it done really fast?” (where “fast” is vaguely defined according to product political circumstances and never puts any weight on engineering estimates, staffing needs, or resource limits) and “did you get it done cheaply?” (where “cheap” is defined vaguely based on various internal politics and budget turf wars as well as larger company financials - and when the money is good and no one looks too hard at this, it allows sweeping other issues under the rug, no one cares if you burnt a quarter working on the wrong problems). Assessing effectiveness, in principle, is purely a political concept that operates from the top down. This has been true in every company I’ve worked for or knew a colleague or friend who worked there - from tiny “lean” startups to extreme cultures like Bridgewater to every mid-sized or large tech company.

dacracot 6y ago

> Just as a sports team wins or loses together, so too should the engineering team be treated as the fundamental unit of success. A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? Too many times I have see engineering teams as only a team on the org chart In reality they solve tickets as individuals with only a small interaction from pull requests. Otherwise they might as well not even know each other. They are a team not as in basketball or football, but like golf where once you get to the tee, it's you and only you to get the ball in the hole.

babesh 6y ago
Baseball
cambalache 6y ago
Not only that but as every sports fan know the compensation in sports team vary tremendously, even for similar "roles". People in tech would be shocked (with reason) if that were the case in the IT world.
voodootrucker 6y ago
> A sports team has a play book, does your team? A sports team practices together, does your team? A sports team works as a unit, does your team? This is why I like XP. Their teams really are teams like you say. Though I think in many dev shops you can be team like. Someone might like refactoring and cleanup. Someone else is good at rapid prototyping. Another architecture. Sometimes a great dev is just the one who can take the unglamorous tickets and get them done at a sustainable pace. Or someone who is good at devops, teaching, or morale building. Sometimes just communication. Everyone has different strengths. No one would ever say "who's the best (American) football player?" because you'd have to ask "who's the best kicker, tight-end, defensive lineman". They are all different roles. To think that football would have the level of awareness that it cannot be measured as a single dimension makes it sad and laughable that people reduce programming skill down to one most of the time.
- lazyasciiart 6y ago

rajacombinator 6y ago

Let’s see an article about measuring manager productivity.

wuweipractioner 6y ago
IMO a manager's productivity is measured based on how well their team performs(or how well they present their team's productivity) and this article provides an approach on how to do so.
- prepend 6y ago
  So the more blockers a manager removes, the better the look to their manager. It’s turtles all the way up. I actually think that one’s compensation is correlated to how ambiguous the job duties are and how hard it is to measure success, given employment. Stuff that’s easy to measure gets commoditized. Stuff that’s hard to measure, but still important is hard to staff, creates risk and is easy to mitigate with money.

mbrodersen 6y ago

The best engineers don’t do much work. They are excellent at delivering the software needed to solve a problem fast and without problems. The worst engineers are always super busy. Fighting fires, starting new fires, rewriting code that doesn’t require rewriting, introducing new shiny tech that will break things in new busy work creating ways etc. etc.

purplecats 6y ago
the latter get promoted and favored by management. go figure.

obiefernandez 6y ago

> If your team is full of competent, driven engineers, removing their blockers is the fastest way to enable forward movement. That's a HUGE if right there. In my experience, the most problematic teams I've ever worked with were simply full of incompetent and/or unmotivated engineers, and for whatever reason it was impossible to replace them.

prepend 6y ago
I think the challenge with this is that blockers only get removed once and super awesome teams preemptively remove blockers before they block. So while it’s good to remove obstacles and maybe helpful to measure if obstacles exist that’s bad, it doesn’t mean much when no obstacles exist and none have been removed in a long time. Maybe they are super awesome and chugging along from all those removed obstacles. Maybe they are super sucky because they don’t see things as obstacles.

sinuhe69 6y ago

If you only measure blockers as an index for productivity then two teams with the same number of blockers should always have the same productivity, regardless how the people are? I cannot fathom how wrong this “measurement” can be. Yes, reducing blockers is important but that is not be the only thing the world cares about.

didibus 6y ago

> Each engineering team is unique, so its blockers will be specific. It's not so simple as "more maker time is better." If your engineering team is new or temporarily misaligned on key goals, more meeting time might be the answer. What never changes is the need for measurement and well-considered, deliberate decisions And we came full circle back to measuring a teams productivity being an art form. Maybe you have a metric now: Hours spent in meetings per week, but nobody knows how much is too much and how little is too little. How do you measure the impact of more meeting on productivity? Or the impact of less meeting? If your measure of productivity is "time in meeting"? This is a circular dependency.

ec109685 6y ago
That’s why they suggest surveys to gauge and looking at how much “maker” time engineers have.

didibus 6y ago

One thing that wasn't covered here is that productivity is about: > the effectiveness of productive effort and > the state or quality of producing something Those are quoted from the dictionary definition of productivity, and that definition in my opinion outlines a great insight. Productivity is about the "product" first and foremost. One thing that's often missing is teams don't have ways to quantify how good the product itself is. Most teams will instead pivot to trying to measure their rate of change to the product. This doesn't mean you have become more productive, because software product is not like food production, more of it is not always better. Better software will instead be about it being more ingenious, more intuitive, more tailored to the problems of its users, more responsive, with fewer malfunctions, etc. So to me, this whole thing of trying to measure "productivity" which disregards the product from the equation is incomplete. It's trying to measure developer efficiency at changing things, without care if the changes are for better or worse. This includes things like measuring velocity, or line of code, or ticket closed. But it also includes what this article proposes, number of meetings, time to complete code reviews, developer satisfaction with their tools, etc. All these are trying to see how quickly can developer make changes without ever measuring if the change produces a better product, thus if it actually made the team more productive at effectively positively improving the product. I'd like to hear about ways to measure software product state and quality. If we had those, you'd have an easy way to know how productive a team would be by seeing how quickly they can improve the product.

kqr 6y ago
Yes! There's a strong tendency toward measuring unvalidated proxies because they are easy or convenient to measure, without even constructing a hypothesis about how these proxies relate to more important things, much less testing it. Measuring productivity has to start with what matters: do your end-users (these are not necessarily your customers) like what you do? And often it is hard to get this information. Just asking them will lead to biased responses, but there are ways to deal with that. I happen to be in a business (let's call it food) where I can observe the end user using more of their money with our customers rather than their competitors when we do things right. That's a strong signal -- probably stronger than asking them -- so a very fortunate starting point. (Of course, there are still confounding terms like seasonality, economy etc to grapple with, but there are ways to deal with that too.) Starting with that one measurement that matters, one can begin exploring proxies. Set up a hypothesis: "Velocity would be easier to measure and the number would be available faster. Does it correlate with our one good metric?" And then you run the experiment. It might take weeks or months to get back the end-user-happiness data that corresponds to this week's velocity, so these tests are expensive (but pay off many times over when you find good proxies. In the end, you should be able to construct a somewhat sensible model of user happiness, and answer questions like, "if we hire another team member and therefore increase velocity by 3 %, how much happier will our users be? And what is that worth in sales?" When you can convert everything to the same unit of measurement (dollars are an easily explained option, but log-dollars are a personal favourite of mine) you get intense clarity and alignment around priorities and decisions. ---- All of that said, development speed, as defined in the Accelerate study in particular, is one of those generally good things you pretty much unconditionally want. The reason is given in the study and expanded on further in Reinertsen's Principles of Product Development. The reason speed is important is that successful product development is controlled by surprises. You will discover something tomorrow that will make you wish you had prioritised differently today, and being able to pivot quickly on surprises is how you both de-fang the biggest risks, but also how you throw yourself at opportunities before your competition even realises there is one.

caseyross 6y ago

One thing I really like about this approach is that it creates strong incentives for engineers to write maintainable code. Code that is expedient but brittle or hard to understand will be noted as blocking progress for future changes, while robust code can be promoted as blocker-free.

vasco 6y ago

The idea of identifying and removing a bottleneck isn't new. I don't know about randomly removing "blockers", it's generally widely accepted that only the bottleneck makes sense improving, everything else is wasted work as you'll just accumulate work in progress. People sometimes rebel a lot against this because "my code is a creative process and that's for factory lines" but I've never found myself worse off when improving a bottleneck, though you do have to fight the "but everything counts" crowd.

GizmoSwan 6y ago

Good work. That is how it should be measured. In economics as far as I know, they measure productivity via salaries. The more money people make, the more productive they are assumed to be!

GizmoSwan 6y ago
'Evaluations of an economy's productivity performance are made using a measure of real gross domestic product (GDP), which represents the constant dollar income (labour income plus profits) that an economy generates through domestic production, with the volume or constant dollar indices being calculated from the prices' 'DP=(GDP/Hours)*(Hours), (1) where Hours is the total number of worker-hours. Chart 1 depicts changes in each of these components over time. For the entire 1961-to-2012 period, labour productivity advanced at a 1.9% annual average, accounting for slightly more than half of the increase in GDP growth. The rest is attributed to hours, which increased at 1.5% per year on average. Aggregate GDP measures the returns to both labour and capital. Distributional concerns lead to questions about whether the share going to labour increases over time and, in particular, how productivity growth is related to real income.' https://www150.statcan.gc.ca/n1/pub/15-206-x/15-206-x2014038-eng.htm https://www150.statcan.gc.ca/n1/pub/15-206-x/15-206-x2014038...

nitwit005 6y ago

I have to disagree with the idea that sales is somehow becoming a scientific process due to CRM. There are openly gut feeling fields like how a call seemed to go, how "hot" an opportunity is, etc. Most of the ideas predate software as well. People had remarkably complex paper based sales systems.

prepend 6y ago
I’m not in sales but I remember my firm having sales pipeline meetings 20 years ago where sales people would talk about stuff that happened. It was all manual and anecdotal. So the sales pipeline was only in someone’s head or maybe a spreadsheet they hacked out once a quarter or whatever. I think the difference today that the author referenced is that with Salesforce stuff like number of leads, contacts, conversion, revenue and profit per conversion, etc is all captured and reportable in sort of real-time. There’s still lots of gut inputs like how the meeting went or having people estimate probability of closure, etc. But now it’s more systematic. I don’t know if it’s more successful, but it seems like I get more sales calls and repeat contact attempts.

marcus_holmes 6y ago

I don't actually see a productivity measure in all this, though. If there are more blockers, is the team doing better (because they're covering ground faster and finding new blockers faster). Or worse (because they're being blocked more)? If I have to report to the CEO on the Dev team productivity, do I tell them how many blockers we removed?

simon_000666 6y ago

or... Just use the metrics outlined in the accelerate book, which have been fairly well studied and have quite a lot of quantifiable evidence to back them up. E.g. Deployment frequency Lead time for changes MTTR Change failure rate A quick google reveals a fair amount of existing material : https://leadingagileteams.com/2020/04/07/forget-dumb-productivity-measures/ https://leadingagileteams.com/2020/04/07/forget-dumb-product...

wiz21c 6y ago

FTA > Productivity in engineering therefore naturally increases when you remove the blockers getting in the way of your team. The guy just discovered the job of the PM or what ?

NumberCruncher 6y ago

I am surprised what a discussion an article can kick of which sole purpose is generating views for people.ai...

NotPavlovsDog 6y ago

> Instead of measuring some approximation of engineering output, software teams should measure actual, observable metrics that directly correlate to effectiveness. I prefer to measure value created and partially tie it to compensation, directly and indirectly. For a dev team for a trading platform, we had an index of income created by product, approved by team and stakeholders, that affected total compensation paid to team. This is a specific example. The general principle is let tech teams make decisions and compensate them for value created, at least partially. Otherwise, when you don't want to share your money, "measurement of productivity / effectiveness" comes in. Because when you measure money, people ask, "why am I not getting more when I make you more"? But if you're not measuring money, why do a business?

TheOtherHobbes 6y ago
Because measuring money is hard and it's not easy to apportion credit. Person A has an interesting-but-not-exceptional idea and gets Person B to fund it. People C, D, and E code it. For most interesting-but-not-exceptional ideas you could replace all of these people with others. If you actually ran this an experiment in a Monte Carlo-but-real kind of a way, you'd find some collaborations would do well, some would do badly, some would fail completely, a few might explode (in a good way). How do you quantify the value of the relative contributions?
- NotPavlovsDog 6y ago
  We've run a variety of experiments that worked for us. What is important is implementing the question in a way that works for the organization. How shall we quantify value, and how much of that index will affect compensation. It's answering the question "why do some people get paid more" in a transparent way.
michaelt 6y ago
> I prefer to measure value created and partially tie it to compensation, directly and indirectly. How does this apply to team that maintains your source control and continuous integration systems?

prepend 6y ago

I like that the article kind of gives up metricing and just says each team should figure it out. This is kind of similar to the no metrics he proposes. If I had to choose, I’d choose no metric over lines of code or number of tickets, etc. I think the issue is that when orgs get big enough they have lots of teams and having some comparability is useful to find lessons learned to spread among teams, etc. I don’t know of any metric that is truly objective for figuring out high productivity teams or individuals just by using it. I think there are some “vital signs” that you want projects to have, but don’t want to fixate on the actual value. Like you don’t care if someone’s pulse is 70 or 80 or 90, but you want to make sure pulse is checked. I think having reviews in git is helpful and not having any is something to look into. I think having contributions from other teams is a good sign, although absence isn’t necessarily bad. I think the positive is by showing how others are finding and asking questions or reviewing or contributing material so that’s probably reuse. I think encouraging information sharing through lunch and learn presentations is good, but is delicate to avoid gaming from people just “making the circuit.” Having an automated CI/CD is a good sign and if code is making it to prod without one, it requires looking into. Theoretically a healthy team will have all these signs, but you could have an awesome team with low numbers and a terrible team with high numbers. So these metrics wouldn’t be useful for detecting productivity and comparing across teams, but would be good for just finding big, lurking problems. I comically work in an org where there are whole teams not using source control, so the “only I can measure myself, give me more money” is a very real challenge.

fukmbas 6y ago

It's such a flawed assumption that engineers must "produce something". Engineers are not hourly labor. Experts must be retained simply for reference.