5 ms·
Kudos to the author for being honest about the flaw in this metric: > It’s ridiculously easy to cheat this metric. Even if you correctly categorize your muda—i
by Aurornis 3y ago
Kudos to the author for being honest about the flaw in this metric:
> It’s ridiculously easy to cheat this metric. Even if you correctly categorize your muda—it’s very tempting to let edge cases slide—all you have to do is stop fixing bugs, defer some needed upgrades, ignore a security vulnerability... and poof! Happy numbers. At a horrible cost.
> Actually, that’s the root of my org’s current capacity problems. They weren’t cheating a metric, but they were under pressure to deliver as much as possible. So they deferred a bunch of maintenance and took some questionable engineering shortcuts. Now they’re paying the price.
> Unfortunately, you can get away with cheating this metric for a long time. Years, really. It’s not like you cut quality one month and then the truth comes out the next month. This is a metric that only works when people are scrupulously honest, including with themselves.
I've had the same experience with well-meaning productivity metrics collection: Even the execs who were really trying to do the right thing would accidentally invent a new metric that looked fantastic for a couple years, then later collapsed when everything else caught up. By then it might be someone else's job to clean up the mess.
Like the author said, this difficulty in this problem is that it can go on for years. If you have an executive team that actually knows how to balance the metrics with the things that are harder to tracked, it might not be a problem. However, as soon as you get an executive looking to game the system for a couple years before jumping to another job, it becomes a doomed venture.
- godelski 3y agoI think most people forget that most metrics are easy to hack and all metrics are hackable (yes, even if you use math & science). So the only rational thing to do is be aware and acknowledge the limitations of your metric and be open about that and what context they operate under. If you aren't aware of the underlying assumptions, your metric is borderline useless. It's like the old saying "there's two types of code: those with bugs and those that no one uses." All systems are buggy and if you aren't looking for them you'll get bit. And in great irony, it is why metrics are the downfall of any meritocracy.
- pixl97 3y agoAll models are wrong, some models are useful.
- bravura 3y agoI'm surprised the author, given the thoughtfulness of much of the post, didn't decide to model tech debt, since then accumulated debt would be quantified frequently (if inaccurately). Especially since they are optimizing for value-add work versus paying down tech debt, which the author acknowledges is what got the engineering org into a paying down tech hole that they are currently mired in before the author's tenure. For this reason, accumulated tech debt should be modeled in their bar charts too. This is far preferable to letting tech debt grow and fester for several years. In a way, I find this possibility almost a regression by the author and possibly irresponsible. I.e. they didn't patch the underlying core issue (rest of org not understanding or assessing tech debt as a liability), and instead push a quantitative measure that actively encourages taking on unmeasured tech debt.
- robertlagrant 3y agoIsn't that the "muda"?
- suprjami 3y agoHow do you quantify technical debt? It doesn't seem possible to track percentage of code as debt. You can't put a comment on code and call this block either debt or good. One way could be to measure different activities. If you spend time solving a lot of bugs (muda) then where are those bugs? If they're in the same old code, does that code need rewritten? If they're in new code, is that code being written in a good way? Can better testing catch these? "Technical debt" seems as hard to measure as developer productivity. It seems you need something you can actually put on a bar chart to measure the effects of having technical debt.
- nevdka 3y agoYou need a way to assign technical debt to a process or practice, then you can evaluate the cost of that process over time. Could be something like... Our team spent 500 hours fixing bugs introduced during overtime, and we spent 250 hours of overtime to introduce those bugs in the first place. So an additional hour of overtime will likely introduce 2 hours of technical debt. Or maybe... Code written using inheritance hierarchies creates 40 minutes of muda per hour, but using composition only creates 20 minutes. Or... every hour spent on throw-away prototypes reduces muda by 2 hours.
- gopher_space 3y agoEducation as a domain has identified key performance metrics for teacher evaluation and has come up with a method of collection they can point to which always results in useful data. It's also an expensive and time-consuming process. You're either burning a stack of money because you absolutely need that info, or a slightly smaller stack to cook the books in full view of people who understand statistics way more than you do.
- draebek 3y ago> Education as a domain has identified key performance metrics for teacher evaluation and has come up with a method of collection they can point to which always results in useful data. Can you please elaborate on this? My impression was that judging teachers is actually fairly hard. See, for example, push-back on standardized testing. The most useful metric I can imagine is where the students are in some number of years, but the obvious (to me) problems are 1. Takes years 2. Lots of confounding factors
- gopher_space 3y agoOn-the-job observations done by someone with way more experience. The kpis that get pushback are indicators of whatever you want them to be. They’re just as indicative of poor government planning or shitty parents.
- fardo 3y ago> On-the-job observations done by someone with way more experience This feels like it puts a potential hard cap on quality growth by discouraging mixups or experimentation that might improve education, but wouldn’t please an old-guard for one reason or another, and discourages alternative class styles which the judge doesn’t approve of. Both of those seem like potentially serious problems in education, given that its structure has with few exceptions been effectively stagnant over the last several hundred years. You therefore may be mistaking “evaluating the success at implementing the widely accepted method” for an “evaluation of quality”.
- asdfman123 3y ago> In other words, in the absence of RoI measures, the percent of engineering time spent on value-add activities is a pretty good proxy for productivity. It's a lagging indicator of productivity. If the team is consumed with fixing old bugs, it means that teams weren't actually that productive years back and were instead just exchanging "results" for technical debt. Instead of creating a perverse incentive to create messes, what if they did something like create a ratio: "60% of work is new features, and 40% is maintenance"? And if maintenance becomes all consuming, could they look back and identify the problems with previous launches?
- DenisM 3y agoDid that ever work for anybody? It’s an obvious idea, someone must have tried it by now.
- weinzierl 3y agoNot sure if that is what you mean, but at Microsoft they have the *Bug Cap Formula, which is: Bug Cap = # engineers * 5 If the bug count exceeds the cap at the end of the sprint, the next sprint is used for bug fixing only.