4 ms·

Most managers won't agree to the types of metrics that you say. Yet most will agree to more arbitrary metrics, like much of what makes up Agile/Scrum. Many man

by p4wnc6 11y ago

Most managers won't agree to the types of metrics that you say. Yet most will agree to more arbitrary metrics, like much of what makes up Agile/Scrum. Many managers do this intentionally, because the greater degree of control over metrics wielded by the manager, the greater degree of arbitrary political manipulation which can be inserted, and the greater freedom to pretend after the fact to have always supported only winning ideas and always opposed only losing ideas all along. Real metrics force managers to be accountable for their true beliefs, which is often what the company needs most and what managers will avoid most.

karmelapple 11y ago
What is a good metric? How can we come up with good metrics?
- jdale27 11y ago
  What is a good metric? My take: a good metric is one that is correlated with customer happiness and can't be gamed by engineers.
  - deleted 11y ago
    [deleted]
  - kyllo 11y ago
    No such thing. You can game any metric.
    - malka 11y ago
      Only if you know that such a metric exists
      - kyllo 11y ago
        That's the whole point of the parent article. People are really good at figuring out what the real metrics are (even when management says otherwise), by simply observing what gets rewarded and what doesn't.
- Consultant32452 11y ago
  I feel like a good metric is how subjectively valuable the person's teammates feel that person is to the team when answering the question anonymously. Being socially awesome will only get you so far, eventually everyone on your team will realize you're not pulling your weight. And team members have a lot more visibility to a person's value than managers.
- p4wnc6 11y ago
  There will always be political deception in the debate over choosing metrics. But one thing that indicates a good management structure is that after metrics have been chosen, there is a very strict and quantitative process for holding people accountable for the metric. In things like Agile/Scrum, it's usually the opposite. Whatever the metrics are doesn't matter. What matters is that the process of holding people accountable is intentionally not consistent or quantitative. One week some burndown graph is the end of the world for your team because some manager was cranky. The next week, a different team with the same burndown graph is lauded as extraordinary and hard-working. It's all about giving managers the maximum possible surface area from which to cherry pick metrics that suit their changing political circumstances. It's hardly at all about actually having an accurate measure of productivity or adherence to timelines. I would argue that this means the subjective debates about metrics aren't very important. Pick some metrics that other people have used, even if they are crappy. Hell, even use Agile/Scrum metrics. But whatever metrics you do use, you have to create a bulletproof way of holding people accountable for them. If doing this leads to bad business outcomes because the metrics weren't good enough, then you can iterate on the metrics themselves and bring them more into alignment with your business constraints. But you can't do anything if the process of holding people accountable is subjected to significant political manipulation. One of the best tools for this is the use of internal prediction markets. Employees must "buy shares" in the project outcomes they believe will be true (e.g. "we will solve problem X by the stated timeline Y and under budget Z"). If you commit to these timeline and budget metrics, say, and you "buy shares" in this business outcome, then you only get paid if the outcome happens. It doesn't matter if some unforeseen event prevented it (you should have impounded the probability of that unforeseen event into your model for the price you are willing to pay for a share of the business outcome). In this manner, people are held accountable for their true beliefs. DARPA and some defense contractors have done experiments on using these to govern expensive military contracts. What is amazing is that often the experiments show that projects governed by prediction markets tend to succeed much more frequently, they tend to be completed fully rather than only partially meeting the required specs, and even after being finished faster and more completely, they also come in under budget more often. And the reaction to the use of prediction markets within the companies where it has been tried: outrage and complete refusal to ever work with them again. Managers become so angry that they don't have the option to engage in political manipulation, to secure blame insurance and make sure they get paid regardless of the business outcome, that they outright refuse to use this technique which is overwhelmingly positive for the business bottom line. There have also been lots of studies showing that prediction markets are very robust against rogue actors or people who attempt analogues of "insider trading" and so forth -- rather than this stuff sabotaging projects, it actually creates a free lunch for other internal stakeholders, and is quickly defeated by the internal market. Of course, these types of markets are only as good as the highest level managers who oversee their internal use, so they are not magic solutions or anything. But the point is that this method exists, and has been shown to work well, and all it costs is for managers to give up their ability to game systems with political manipulation. And that cost appears to induce outrage in most managers, who are unwilling to give up political manipulation in exchange for truly better overall business outcomes.
Spooky23 11y ago
There's another angle as well at play... My command chain cares about delivery of services and products at our below the budget, period. But there are others with other interests. Program/project methodology religious adherents. Batshit crazy bureaucrats. All of these people want to insert stuff to serve whatever agenda. I want to deliver on my commitments. Letting people without a clue evaluate my team's effectiveness is not in my team's interest or mine.
- p4wnc6 11y ago
  > There's another angle as well at play... My command chain cares about delivery of services and products at our below the budget, period. This is classic duplicitous manager babble that really just means, "I want authority to decide arbitrarily when we should adhere to bureaucratic rules or project management methodology (... or good coding practices, or sane deadlines, or ...)." When people actually care about the business bottom line (what you describe as delivery of services and products at [or] below the budget), they tend to behave rationally and expand their point of view to include things like bureaucratic necessity, the importance of strictly adhering to chosen project management and code quality standards and never cherry picking which business concerns are supposed to override them, and so forth. When people up the command chain don't behave rationally about this stuff, it's generally because they are bad at their jobs. Bad managers see subordinates as work receptacles who should "just get shit done" and "solve my problems" without any understanding as to how it works or what constraints from reality actually mean in terms of unpackaging business priorities and mapping them to actions to accomplish them. As a manager caught in the middle, you can chose which way you want to go. Appease the batshit bureaucrats by hammering down on your own subordinates, bottomlessly compromising away standards all in the name of so-called business bottom line. Or stand up to them and explain what it takes to actually deliver products. If you lose your job doing this, it's a good thing! It allows you to rejection sample bad employers and more quickly find good employers where the managers realize that the phrase "business priorities" has to be recursively unpacked into all the messy details of project management, budgets, deadlines, work/life balance, and so on, and who actually care to do so (and yes, they really, really exist, just not embedded in brainfucked corporate/organizational hierarchies). One example from my own career: I worked on a team of developers for an education technology company. The company acquired a start-up that had created a suite of learning applications. The code was an unbelievable mess: written in a (bad) proprietary language, poor anti-patterns all over, no usage of proper version control or issue tracking, no sane build or deployment system. To boot, the apps themselves consistently got poor reviews from users in all of the app stores and review sites. And instead of fixing the app and improving the project management methods (truly the only two options for even short term profitability of the app), the management decided to essentially throw gas on the fire and pursue a strategy similar to regulatory capture, and attempt to make the creation phase of the learning materials (things only professors, teachers, or administrators would do) into the first-class constructs of the whole system, further crapifying the experience and functionality for end users (students) and driving app engagement and reviews even lower. After a while of this utterly not working, there were several heart to heart meetings between engineers, middle management, and senior management, at which point it came to light that one of the main reasons we were even pursuing this strategy at all is that there was somewhat of a dick-measuring competition going on between the company's CTO (who was in charge of this portfolio of products), and the CEO of the start-up that had been acquired. Apparently the folks in that start-up, once their golden handcuffs contracts ended, basically took most of the same e-learning idea and tried to spin it to sell it for corporate learning (like corporate training videos and stuff). And our CTO, who was under pressure about the whole deal and our continued poor app ratings, took it personally and basically started enforcing business strategy solely to "stick it" to that startup CEO and compete directly with him for a new market (corporate learning) that we weren't set up to service and had no prior ties into and had no infrastructure to deal with. And of course, every middle manager in between was feeding all of us engineers the same line of bull as from your comment. Everything was just "upper management cares about most efficient delivery of our product" -- when it was patently obvious that they did not at all care about this and that we were being reprioritized constantly to service the arbitrary whim of the CTO. And because this was a huge organization with tens of thousands of employees, consistently losing tens of millions of dollars on a monthly basis to competitor apps who actually took software quality seriously didn't even register as a blip to worry about. If our whole team got laid off because this CTO's strategy of irrational pissing contest competition didn't pan out, who cares? We'll just slap some crap on the front about how the market is changing or we need to streamline or restructure or whatever. My main takeaway was basically that any time a middle manager tells you a decision is being made because it is important for the efficiency of reaching the company's bottom line goals, that manager is just spewing bullshit. When that stuff really happens, non-bullshit-spewers will actually sit down with you and explain it, in nitty gritty detail, and will actually listen to and value your input about whether it's a good idea or not. They won't rush to label you as a "batshit bureaucrat" or a "religious adherent" of quality practices. They'll just listen and be open and transparent.

by p4wnc6 11y ago

karmelapple 11y ago

What is a good metric? How can we come up with good metrics?

jdale27 11y ago
What is a good metric? My take: a good metric is one that is correlated with customer happiness and can't be gamed by engineers.
- deleted 11y ago
  [deleted]
- kyllo 11y ago
  No such thing. You can game any metric.
  - malka 11y ago
    Only if you know that such a metric exists
    - kyllo 11y ago
      That's the whole point of the parent article. People are really good at figuring out what the real metrics are (even when management says otherwise), by simply observing what gets rewarded and what doesn't.
Consultant32452 11y ago
I feel like a good metric is how subjectively valuable the person's teammates feel that person is to the team when answering the question anonymously. Being socially awesome will only get you so far, eventually everyone on your team will realize you're not pulling your weight. And team members have a lot more visibility to a person's value than managers.
p4wnc6 11y ago
There will always be political deception in the debate over choosing metrics. But one thing that indicates a good management structure is that after metrics have been chosen, there is a very strict and quantitative process for holding people accountable for the metric. In things like Agile/Scrum, it's usually the opposite. Whatever the metrics are doesn't matter. What matters is that the process of holding people accountable is intentionally not consistent or quantitative. One week some burndown graph is the end of the world for your team because some manager was cranky. The next week, a different team with the same burndown graph is lauded as extraordinary and hard-working. It's all about giving managers the maximum possible surface area from which to cherry pick metrics that suit their changing political circumstances. It's hardly at all about actually having an accurate measure of productivity or adherence to timelines. I would argue that this means the subjective debates about metrics aren't very important. Pick some metrics that other people have used, even if they are crappy. Hell, even use Agile/Scrum metrics. But whatever metrics you do use, you have to create a bulletproof way of holding people accountable for them. If doing this leads to bad business outcomes because the metrics weren't good enough, then you can iterate on the metrics themselves and bring them more into alignment with your business constraints. But you can't do anything if the process of holding people accountable is subjected to significant political manipulation. One of the best tools for this is the use of internal prediction markets. Employees must "buy shares" in the project outcomes they believe will be true (e.g. "we will solve problem X by the stated timeline Y and under budget Z"). If you commit to these timeline and budget metrics, say, and you "buy shares" in this business outcome, then you only get paid if the outcome happens. It doesn't matter if some unforeseen event prevented it (you should have impounded the probability of that unforeseen event into your model for the price you are willing to pay for a share of the business outcome). In this manner, people are held accountable for their true beliefs. DARPA and some defense contractors have done experiments on using these to govern expensive military contracts. What is amazing is that often the experiments show that projects governed by prediction markets tend to succeed much more frequently, they tend to be completed fully rather than only partially meeting the required specs, and even after being finished faster and more completely, they also come in under budget more often. And the reaction to the use of prediction markets within the companies where it has been tried: outrage and complete refusal to ever work with them again. Managers become so angry that they don't have the option to engage in political manipulation, to secure blame insurance and make sure they get paid regardless of the business outcome, that they outright refuse to use this technique which is overwhelmingly positive for the business bottom line. There have also been lots of studies showing that prediction markets are very robust against rogue actors or people who attempt analogues of "insider trading" and so forth -- rather than this stuff sabotaging projects, it actually creates a free lunch for other internal stakeholders, and is quickly defeated by the internal market. Of course, these types of markets are only as good as the highest level managers who oversee their internal use, so they are not magic solutions or anything. But the point is that this method exists, and has been shown to work well, and all it costs is for managers to give up their ability to game systems with political manipulation. And that cost appears to induce outrage in most managers, who are unwilling to give up political manipulation in exchange for truly better overall business outcomes.

Spooky23 11y ago

There's another angle as well at play... My command chain cares about delivery of services and products at our below the budget, period. But there are others with other interests. Program/project methodology religious adherents. Batshit crazy bureaucrats. All of these people want to insert stuff to serve whatever agenda. I want to deliver on my commitments. Letting people without a clue evaluate my team's effectiveness is not in my team's interest or mine.

p4wnc6 11y ago
> There's another angle as well at play... My command chain cares about delivery of services and products at our below the budget, period. This is classic duplicitous manager babble that really just means, "I want authority to decide arbitrarily when we should adhere to bureaucratic rules or project management methodology (... or good coding practices, or sane deadlines, or ...)." When people actually care about the business bottom line (what you describe as delivery of services and products at [or] below the budget), they tend to behave rationally and expand their point of view to include things like bureaucratic necessity, the importance of strictly adhering to chosen project management and code quality standards and never cherry picking which business concerns are supposed to override them, and so forth. When people up the command chain don't behave rationally about this stuff, it's generally because they are bad at their jobs. Bad managers see subordinates as work receptacles who should "just get shit done" and "solve my problems" without any understanding as to how it works or what constraints from reality actually mean in terms of unpackaging business priorities and mapping them to actions to accomplish them. As a manager caught in the middle, you can chose which way you want to go. Appease the batshit bureaucrats by hammering down on your own subordinates, bottomlessly compromising away standards all in the name of so-called business bottom line. Or stand up to them and explain what it takes to actually deliver products. If you lose your job doing this, it's a good thing! It allows you to rejection sample bad employers and more quickly find good employers where the managers realize that the phrase "business priorities" has to be recursively unpacked into all the messy details of project management, budgets, deadlines, work/life balance, and so on, and who actually care to do so (and yes, they really, really exist, just not embedded in brainfucked corporate/organizational hierarchies). One example from my own career: I worked on a team of developers for an education technology company. The company acquired a start-up that had created a suite of learning applications. The code was an unbelievable mess: written in a (bad) proprietary language, poor anti-patterns all over, no usage of proper version control or issue tracking, no sane build or deployment system. To boot, the apps themselves consistently got poor reviews from users in all of the app stores and review sites. And instead of fixing the app and improving the project management methods (truly the only two options for even short term profitability of the app), the management decided to essentially throw gas on the fire and pursue a strategy similar to regulatory capture, and attempt to make the creation phase of the learning materials (things only professors, teachers, or administrators would do) into the first-class constructs of the whole system, further crapifying the experience and functionality for end users (students) and driving app engagement and reviews even lower. After a while of this utterly not working, there were several heart to heart meetings between engineers, middle management, and senior management, at which point it came to light that one of the main reasons we were even pursuing this strategy at all is that there was somewhat of a dick-measuring competition going on between the company's CTO (who was in charge of this portfolio of products), and the CEO of the start-up that had been acquired. Apparently the folks in that start-up, once their golden handcuffs contracts ended, basically took most of the same e-learning idea and tried to spin it to sell it for corporate learning (like corporate training videos and stuff). And our CTO, who was under pressure about the whole deal and our continued poor app ratings, took it personally and basically started enforcing business strategy solely to "stick it" to that startup CEO and compete directly with him for a new market (corporate learning) that we weren't set up to service and had no prior ties into and had no infrastructure to deal with. And of course, every middle manager in between was feeding all of us engineers the same line of bull as from your comment. Everything was just "upper management cares about most efficient delivery of our product" -- when it was patently obvious that they did not at all care about this and that we were being reprioritized constantly to service the arbitrary whim of the CTO. And because this was a huge organization with tens of thousands of employees, consistently losing tens of millions of dollars on a monthly basis to competitor apps who actually took software quality seriously didn't even register as a blip to worry about. If our whole team got laid off because this CTO's strategy of irrational pissing contest competition didn't pan out, who cares? We'll just slap some crap on the front about how the market is changing or we need to streamline or restructure or whatever. My main takeaway was basically that any time a middle manager tells you a decision is being made because it is important for the efficiency of reaching the company's bottom line goals, that manager is just spewing bullshit. When that stuff really happens, non-bullshit-spewers will actually sit down with you and explain it, in nitty gritty detail, and will actually listen to and value your input about whether it's a good idea or not. They won't rush to label you as a "batshit bureaucrat" or a "religious adherent" of quality practices. They'll just listen and be open and transparent.