5 ms·

>I’m not convinced that related behaviors, like constructing a game adjacent to the problem you’re actually trying to solve and treating game performance as a p

by na85 5y ago

>I’m not convinced that related behaviors, like constructing a game adjacent to the problem you’re actually trying to solve and treating game performance as a proxy (like LeetCode interviews! Or KPIs!) are helpful. The devil is as always in the details. At least as far as KPIs go, they can be very useful if they are chosen appropriately and if the senior team members have all bought in to the concept. Example: I used to work in military search and rescue, and the pilots get tasked by a complex bureaucracy indirectly descended from NORAD, and they have a requirement in most areas to be airborne within 30 minutes when called. Thus, our maintenance flight had as its first and foremost KPI chosen "percentage of the time where maintenance can't get an aircraft ready to be flying missions in less than 30 minutes". That's a fantastic KPI because the maintainers that worked for us were all on board. Getting that KPI as close to 0 as possible (and we did hit zero every few months) was a goal that everyone had bought into, because of course if you can't put an aircraft in the sky then it potentially means people are dying unnecessarily. Now, any AME or military aircraft tech can tell you the paperwork requirements for aircraft maintenance are very strict and at times quite onerous, but they're required by federal law. So the same maintenance organization also had, broadly, "paperwork error rates" as a tertiary KPI, because of course the maintenance paperwork gets audited by an AS9100 or similar quality assurance shop, usually in-house. That one is a shitty KPI because the wrench turners who work on complex jobs are more likely to make errors in the paperwork, so there's a perverse incentive wherein your best maintainers make about the same number of mistakes as your journeymen right out of school, simply because it's easier to fill out the paperwork properly for topping up an oil reservoir compared to changing a propeller or a weight-and-balance. All that to say KPIs themselves are not the problem. Shitty leaders who choose KPIs improperly are the problem.

kqr 5y ago
I agree with everything you say except the bit where the failure rate is a fantastic KPI. It has room for improvement. > they have a requirement in most areas to be airborne within 30 minutes when called. [...] "percentage of the time where maintenance can't get an aircraft ready to be flying missions in less than 30 minutes". The problem with this KPI is that it's completely arbitrary. Why does it have to be 30 minutes, and not 25, or 35? Changing this threshold might change your failure percentage significantly. This KPI is not actually a ruler that measures your performance – it's using your performance as a ruler to measure the threshold, because the easiest way to change the KPI number is by changing the threshold, not the performance. After all, wouldn't 25 be better than 30? The only reason it's 30 is that someone at some point said it has to be 30. An arbitrary cutoff limit like 30 minutes (what Deming would call a numerical goal without a method) has one of three effects: - Either it's really hard to accomplish, in which case the large failure rate is demotivating, or - It's just about what you can do, in which case it's meaningless anyway, or - It's easier than what you could technically do, which might inspire dilatoriness: "You can't complain; we're meeting our goals!" Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time. I know, I know, the 30 minute number is not entirely arbitrary, it's probably related to how easy it is to keep a person alive. But it's arbitrary in the sense of being a cutoff – for keeping people alive, shorter is always better. There's nothing magical happening at exactly the 30 minute mark.
- thaumasiotes 5y ago
  > Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time. This is an example of a more general phenomenon - when you have outcome buckets, it's generally better to do your modeling on raw outcomes ["it took us 32 minutes"] than to try to do it on the bucketed outcomes ["we didn't make it in 30 minutes"]. Bucketing throws most of your information away. Andrew Gelman talks about this all the time in the context of election modeling, where it's better to predict how many votes something will get, which is a continuous variable (and then, based on the predicted vote, predict whether the thing will win or not), rather than trying to predict whether the thing will win, which is a discrete variable.
  - dmurray 5y ago
    > Use "minutes until airborne" as the KPI. Average minutes to airborne? Median? Mode? 90th percentile? 99th percentile? Optimizing for each of those would result in a different process, different outcomes, and quite possibly worse outcomes for the stakeholders than "proportion under 30 minutes" which is just a sixth way to slice the same data.
    - thaumasiotes 5y ago
      Did you mean to respond to my parent comment?
    - kqr 5y ago
      > Average minutes to airborne? Median? Mode? 90th percentile? 99th percentile? Facetious answer: yes. More useful answer: you want to record the full timeseries of individual data points. From this you can derive mean, median, mode, 90th percentile, 99th percentile and any other statistic that is shaped like hole in your cost–benefit calculations. Including trends and changing variance. For reporting purposes, the upper process behaviour limit is probably a good one as "the" numeric value of the KPI, but the true value is in the timeseries.
tonyedgecombe 5y ago
>Getting that KPI as close to 0 as possible (and we did hit zero every few months) was a goal that everyone had bought into So what corners used to get cut to meet that target? What extra costs were incurred?
closeparen 5y ago
>All that to say KPIs themselves are not the problem. Shitty leaders who choose KPIs improperly are the problem. Deciding to govern by KPIs presupposes that we have KPIs that are chosen well enough to beat subject-matter-expert judgement. I think if we are talking about anything other than a game with very clear rules and observable outcomes, that's an extraordinary claim.

by na85 5y ago

kqr 5y ago

I agree with everything you say except the bit where the failure rate is a fantastic KPI. It has room for improvement. > they have a requirement in most areas to be airborne within 30 minutes when called. [...] "percentage of the time where maintenance can't get an aircraft ready to be flying missions in less than 30 minutes". The problem with this KPI is that it's completely arbitrary. Why does it have to be 30 minutes, and not 25, or 35? Changing this threshold might change your failure percentage significantly. This KPI is not actually a ruler that measures your performance – it's using your performance as a ruler to measure the threshold, because the easiest way to change the KPI number is by changing the threshold, not the performance. After all, wouldn't 25 be better than 30? The only reason it's 30 is that someone at some point said it has to be 30. An arbitrary cutoff limit like 30 minutes (what Deming would call a numerical goal without a method) has one of three effects: - Either it's really hard to accomplish, in which case the large failure rate is demotivating, or - It's just about what you can do, in which case it's meaningless anyway, or - It's easier than what you could technically do, which might inspire dilatoriness: "You can't complain; we're meeting our goals!" Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time. I know, I know, the 30 minute number is not entirely arbitrary, it's probably related to how easy it is to keep a person alive. But it's arbitrary in the sense of being a cutoff – for keeping people alive, shorter is always better. There's nothing magical happening at exactly the 30 minute mark.

thaumasiotes 5y ago
> Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time. This is an example of a more general phenomenon - when you have outcome buckets, it's generally better to do your modeling on raw outcomes ["it took us 32 minutes"] than to try to do it on the bucketed outcomes ["we didn't make it in 30 minutes"]. Bucketing throws most of your information away. Andrew Gelman talks about this all the time in the context of election modeling, where it's better to predict how many votes something will get, which is a continuous variable (and then, based on the predicted vote, predict whether the thing will win or not), rather than trying to predict whether the thing will win, which is a discrete variable.
- dmurray 5y ago
  > Use "minutes until airborne" as the KPI. Average minutes to airborne? Median? Mode? 90th percentile? 99th percentile? Optimizing for each of those would result in a different process, different outcomes, and quite possibly worse outcomes for the stakeholders than "proportion under 30 minutes" which is just a sixth way to slice the same data.
  - thaumasiotes 5y ago
    Did you mean to respond to my parent comment?
  - kqr 5y ago
    > Average minutes to airborne? Median? Mode? 90th percentile? 99th percentile? Facetious answer: yes. More useful answer: you want to record the full timeseries of individual data points. From this you can derive mean, median, mode, 90th percentile, 99th percentile and any other statistic that is shaped like hole in your cost–benefit calculations. Including trends and changing variance. For reporting purposes, the upper process behaviour limit is probably a good one as "the" numeric value of the KPI, but the true value is in the timeseries.

tonyedgecombe 5y ago

>Getting that KPI as close to 0 as possible (and we did hit zero every few months) was a goal that everyone had bought into So what corners used to get cut to meet that target? What extra costs were incurred?

closeparen 5y ago

>All that to say KPIs themselves are not the problem. Shitty leaders who choose KPIs improperly are the problem. Deciding to govern by KPIs presupposes that we have KPIs that are chosen well enough to beat subject-matter-expert judgement. I think if we are talking about anything other than a game with very clear rules and observable outcomes, that's an extraordinary claim.