5 ms·

> This will inevitably shift the company's culture towards gaming those metrics. Yes, but that's only the beginning of the story. People gaming metrics is a ty

> This will inevitably shift the company's culture towards gaming those metrics. Yes, but that's only the beginning of the story. People gaming metrics is a type of security problem, in that "attackers" try to game the metrics while "defenders" try to make them less game-able by improving the accuracy and precision of how the metrics are gathered so that the final numbers continue to tell a valuable story over time. The issue isn't that metrics can be gamed; it's that organizations which pride themselves on being data-driven rarely make the investment in hiring blue teams and red teams to defend and attack the metrics. If you appreciate that investing in cyberdefense is key to protecting your company from cybersecurity threats, why can't you appreciate that investing in "metricsecurity" is key to protecting your company from "metricsecurity" threats?

Angostura 8y ago
> The issue isn't that metrics can be gamed The issue is that many of the most important parts of many organisational activities can't be easily measured through simple metrics at all.
- solatic 8y ago
  I think this is a rubbish excuse made by people who don't understand the role of such a "blue team". If the organization initially adopts metrics that are counter-productive (e.g. measuring feature completion and not technical debt), it is the role of the blue team to change the metrics such that the neglected areas are properly accounted for in final performance metrics. No metrics should be final; only iteratively tuned to achieve results that are more and more indicative of the underlying performance. It is difficult, but still possible, to measure technical debt and other "hard" metrics. It is precisely the job of the blue team to deal with that.
  - hammock 8y ago
    There is a philosophical debate underlying this. Take the analogy of a ML algorithm: We know many algos are DESIGNED to be a black box, to be unexplainable. Red team iterates an incredibly effective algorithm that produces the desired outputs (with unseen risk built up as well). Blue team, in order to manage risk, is tasked with .. explaining the unexplainable process? An impossible task. Is it possible that human/organizational processes can be also unexplainable?
  - rodeoclown 8y ago
    Just because you've made a team and given them the job of doing something (incredibly) difficult, doesn't mean you've actually solved that problem or even should expect them to solve it most of the time. You're absolutely right that having a "blue team" is much better than not having one - but it doesn't mean that calling out the reality that many organizational activities can't be easily measured is a "rubbish excuse made by people who don't understand".
  - Angostura 8y ago
    And how do you measure the person who takes time out of their day to help a colleague in another department, who is having a tough time understanding an issue, so the person takes 15 minutes out to help them, boosting morale and overall company cohesion? Or should they be penalised for wasting 15 minutes?
    - bluGill 8y ago
      Depends, do you want that to happen or not? You seem to be making an assumption this is good, but in some organizations this would be a bad thing to penalize. I'm not sure why you would do this in engineering, but it is important to acknowledge that this isn't a universal good and so maybe your company wants to discourage it for some reason. Assuming you want people to help each other, you need to capture metrics on it. A few years back I had a metric of helping n people in a different department: I kept track of those interactions so I had something to report at the end of the year.
      - TeMPOraL 8y ago
        Was that your personal metrics? Metrics created for yourself are subject to less gaming because when you start lying to yourself about those, you will start to wonder why keep those metrics at all. If that was a company-issued, top-down metric, I hope it wasn't defined literally as "helping n people in a different department", because that has enough wiggle room to sail an aircraft carrier through. The difficulty of creating a good metric here comes from the difficulty of defining what exactly does it mean, in company context, to "help other people" - and also what it explicitly doesn't mean.
        bluGill 8y ago
        I had to report it to my boss. It was top down, but only a few interactions were required, and it wasn't reported farther up the chain. Because I had to report to my boss, he knew me well enough to judge if it was enough. It was just enough of a metric to ensure people looked for something to bridge a communication gap, without being hard enough that people tried to game it much.
  - slavik81 8y ago
    What metric do you use to judge the success of the blue team, and what prevents them from gaming that metric?
    - solatic 8y ago
      In practice, you wouldn't really need to have a metric for the blue team, for the same reason you don't need metrics in a 5-person startup. Management is close to the blue team to replace being "close" to thousands of people, and because management is close to the blue team, can judge their output without needing a formal metric. Maybe if you had an organization that was big enough to require several blue teams (a military or government?), then you'd need a metric for blue teams. Such a metric would probably compare the sub-KPIs of each sub-organization that each blue team was responsible for, including metrics on customer satisfaction, and warrant investigation if the metrics went under. The blue teams can't really game that metric without the entire organization falling over, and if that happened, the executives would be to blame, not the blue team.
      - hammock 8y ago
        I know a girl who works for United Airlines' "blue team." It's a small (5-8 people?) inward-facing operations consulting group that reports directly to Oscar, the CEO. It's composed of engineers and they analyze existing processes and create new metrics all day long. They do worry about their own careers/promotions etc but the group is too small for there really to be any opportunities to "game" anything beyond basic politics.
        pinko 8y ago
        I trust it's just habitual, but I think it's important to describe her as a woman and not a girl. When a company is casually described as being made up of men and girls, it sends an insidious message to everyone about who's taken more seriously.
      - dragonwriter 8y ago
        > In practice, you wouldn't really need to have a metric for the blue team So the metrics are BS, and the company is really being run by subjective intuition, for which th metrics merely provide an impersonal rationalization.
- natalyarostova 8y ago
  The problem though is that executives need some set of a few hundred numbers they can use to track the state of a company. For just my job alone I could generate more metrics then that to properly characterize our state and problem space -- but then an exec would need to deal with thousands and thousands of numbers. Sorta sucks, but that's how it goes. Good execs manage a sufficiently decentralized system, but they still need SOME set of summary numbers.
HelloNurse 8y ago
Good theory, but in practice challenging the wisdom of metrics that really important managers have put into place tends to be a career-limiting move. The very important managers themselves should care about evaluating the metrics they impose, but typical unspoken manager performance metrics include "episodes of dissent" (addressed by discouraging advice), "displays of weakness" (addressed by threats and aggressive attitude), "time management" (addressed by not thinking matters through), and so on.
ABCLAW 8y ago
>why can't you appreciate that investing in "metricsecurity" is key to protecting your company from "metricsecurity" threats? Because if I'm a C-level or EVP-level person responsible for this type of decision, why would I want to spend money on a team of people fighting against my ability to get a big fat bonus?

Angostura 8y ago

> The issue isn't that metrics can be gamed The issue is that many of the most important parts of many organisational activities can't be easily measured through simple metrics at all.

solatic 8y ago
I think this is a rubbish excuse made by people who don't understand the role of such a "blue team". If the organization initially adopts metrics that are counter-productive (e.g. measuring feature completion and not technical debt), it is the role of the blue team to change the metrics such that the neglected areas are properly accounted for in final performance metrics. No metrics should be final; only iteratively tuned to achieve results that are more and more indicative of the underlying performance. It is difficult, but still possible, to measure technical debt and other "hard" metrics. It is precisely the job of the blue team to deal with that.
- hammock 8y ago
  There is a philosophical debate underlying this. Take the analogy of a ML algorithm: We know many algos are DESIGNED to be a black box, to be unexplainable. Red team iterates an incredibly effective algorithm that produces the desired outputs (with unseen risk built up as well). Blue team, in order to manage risk, is tasked with .. explaining the unexplainable process? An impossible task. Is it possible that human/organizational processes can be also unexplainable?
- rodeoclown 8y ago
  Just because you've made a team and given them the job of doing something (incredibly) difficult, doesn't mean you've actually solved that problem or even should expect them to solve it most of the time. You're absolutely right that having a "blue team" is much better than not having one - but it doesn't mean that calling out the reality that many organizational activities can't be easily measured is a "rubbish excuse made by people who don't understand".
- Angostura 8y ago
  And how do you measure the person who takes time out of their day to help a colleague in another department, who is having a tough time understanding an issue, so the person takes 15 minutes out to help them, boosting morale and overall company cohesion? Or should they be penalised for wasting 15 minutes?
  - bluGill 8y ago
    Depends, do you want that to happen or not? You seem to be making an assumption this is good, but in some organizations this would be a bad thing to penalize. I'm not sure why you would do this in engineering, but it is important to acknowledge that this isn't a universal good and so maybe your company wants to discourage it for some reason. Assuming you want people to help each other, you need to capture metrics on it. A few years back I had a metric of helping n people in a different department: I kept track of those interactions so I had something to report at the end of the year.
    - TeMPOraL 8y ago
      Was that your personal metrics? Metrics created for yourself are subject to less gaming because when you start lying to yourself about those, you will start to wonder why keep those metrics at all. If that was a company-issued, top-down metric, I hope it wasn't defined literally as "helping n people in a different department", because that has enough wiggle room to sail an aircraft carrier through. The difficulty of creating a good metric here comes from the difficulty of defining what exactly does it mean, in company context, to "help other people" - and also what it explicitly doesn't mean.
      - bluGill 8y ago
        I had to report it to my boss. It was top down, but only a few interactions were required, and it wasn't reported farther up the chain. Because I had to report to my boss, he knew me well enough to judge if it was enough. It was just enough of a metric to ensure people looked for something to bridge a communication gap, without being hard enough that people tried to game it much.
- slavik81 8y ago
  What metric do you use to judge the success of the blue team, and what prevents them from gaming that metric?
  - solatic 8y ago
    In practice, you wouldn't really need to have a metric for the blue team, for the same reason you don't need metrics in a 5-person startup. Management is close to the blue team to replace being "close" to thousands of people, and because management is close to the blue team, can judge their output without needing a formal metric. Maybe if you had an organization that was big enough to require several blue teams (a military or government?), then you'd need a metric for blue teams. Such a metric would probably compare the sub-KPIs of each sub-organization that each blue team was responsible for, including metrics on customer satisfaction, and warrant investigation if the metrics went under. The blue teams can't really game that metric without the entire organization falling over, and if that happened, the executives would be to blame, not the blue team.
    - hammock 8y ago
      I know a girl who works for United Airlines' "blue team." It's a small (5-8 people?) inward-facing operations consulting group that reports directly to Oscar, the CEO. It's composed of engineers and they analyze existing processes and create new metrics all day long. They do worry about their own careers/promotions etc but the group is too small for there really to be any opportunities to "game" anything beyond basic politics.
      - pinko 8y ago
        I trust it's just habitual, but I think it's important to describe her as a woman and not a girl. When a company is casually described as being made up of men and girls, it sends an insidious message to everyone about who's taken more seriously.
    - dragonwriter 8y ago
      > In practice, you wouldn't really need to have a metric for the blue team So the metrics are BS, and the company is really being run by subjective intuition, for which th metrics merely provide an impersonal rationalization.
natalyarostova 8y ago
The problem though is that executives need some set of a few hundred numbers they can use to track the state of a company. For just my job alone I could generate more metrics then that to properly characterize our state and problem space -- but then an exec would need to deal with thousands and thousands of numbers. Sorta sucks, but that's how it goes. Good execs manage a sufficiently decentralized system, but they still need SOME set of summary numbers.

HelloNurse 8y ago

Good theory, but in practice challenging the wisdom of metrics that really important managers have put into place tends to be a career-limiting move. The very important managers themselves should care about evaluating the metrics they impose, but typical unspoken manager performance metrics include "episodes of dissent" (addressed by discouraging advice), "displays of weakness" (addressed by threats and aggressive attitude), "time management" (addressed by not thinking matters through), and so on.

ABCLAW 8y ago

>why can't you appreciate that investing in "metricsecurity" is key to protecting your company from "metricsecurity" threats? Because if I'm a C-level or EVP-level person responsible for this type of decision, why would I want to spend money on a team of people fighting against my ability to get a big fat bonus?