7 ms·

The times I've seen OKRs and metrics used effectively, they met two criteria. (1) it was always a metric closely aligned with what customers want and which driv

The times I've seen OKRs and metrics used effectively, they met two criteria. (1) it was always a metric closely aligned with what customers want and which drives revenue for the company; and (2) it didn't just impose a requirement to do work, it also provided political cover for not doing other, competing work. For example, if you're designing GPUs and/or GPU drivers? If your next-generation product has the aim of providing 25% more frames per second in "Baldur's Gate 3" and "Call of Duty 6" while maintaining the same quality - that would be a good objective for the team, as it's closely aligned with what your customers want. And if someone should come to you and tell you there's a lot of people streaming these days, and they think you should optimise gaming FPS and h264 compression at the same time? It's a sensible request but it's also a distraction; proper goal setting will let you say "great idea, but not this quarter". But there are a lot of fields of endeavour where it's not possible to meet these criteria - like the suicide prevention team from your example.

ricardobeat 2y ago
100% agree. There is a basic level of honesty required for this to work, which seems to have gone missing due to the "evolution" of corporate culture and the cash-rich environment which made actual productivity a secondary concern.
DanielHB 2y ago
In my job we have metrics, but they are not mandated to employees. There are no targets and stuff like that, they are just there if you want to know. Managers track it and try to optimize for it, but nobody bonuses or employment is on the line over them. Maybe some sales people have commission, but most of the customer acquisition is organic anyway. Funnily enough we have a weird situation right now, where we want to optimize the _cheaper_ plan of our product, meaning moving people from the more expensive plans to the cheaper one. It is a weird dynamic, but due to licensing our cheaper plan is actually far more profitable than the more expensive plans. Most clients wouldn't really miss anything from the more expensive plan. It is one of those rare situations where convincing people to switch to the cheaper plan aligns with what is best for the consumer.
- ozim 2y ago
  Well that sounds like good idea. As a manager you should have metrics because then you know if you tweak the process correctly. Not sharing it with employees and definitely not making any bonuses or pay increases depending on metrics. Because then employees will game your metrics and they will be useless.
aflukasz 2y ago
> For example, if you're designing GPUs and/or GPU drivers? If your next-generation product has the aim of providing 25% more frames per second in "Baldur's Gate 3" and "Call of Duty 6" while maintaining the same quality - that would be a good objective for the team, as it's closely aligned with what your customers want. I can get selection of particular gaming titles, but how do you come up with 25% goal? How is this closely aligned? Your users tell you they want ~30% gains? This seems to be completely ignoring a constant feedback loop between general aspirations for the product, operated timeframes, and conclusions from ongoing engineering R&D.
- exe34 2y ago
  I've never understood that sort of target - managers at McDonald's had them when I flipped burgers through uni, and they would just pull some numbers out of somewhere and get very excited about them. then they would nag customers until they hit those numbers, presumably.
- michaelt 2y ago
  > how do you come up with 25% goal? In organisations that do this kind of work, the 'marketing' department isn't just about placing adverts and writing blog posts. They also have people whose job is to keep track of what's going on in the market, and to estimate what your product needs in order to be competitive when it comes out. To take a simple example, the marketing team might have visited https://www.tomshardware.com/reviews/amd-radeon-rx-7900-xtx-and-xt-review-shooting-for-the-top/4 https://www.tomshardware.com/reviews/amd-radeon-rx-7900-xtx-... and found on 4K ultra settings that the "RX 7900 XTX" shows up as 90.3 fps average and the "RTX 4090" shows up as 112.6 fps average. And 112.6÷90.3 = 1.246 so if AMD want their next-gen flagship to outperform nvidia's current-gen flagship, they need 24.6% more FPS. Of course it's a lot more complicated than that in practice. They'd also consider value-for-money, non-flagship cards, whether users' monitors even support >120fps, guessing at what nvidia's next flagship will offer, interviewing big buyers like Dell, and so on. But that's the general gist of it.
liontwist 2y ago
And then customers complain that the driver is frequently crashing and the team is well aware but won’t fix the ticket because it doesn’t help them meet OKRs. What you really want is: - every team working to improve the customer experience - guided by leadership priorities and initiatives
- madeofpalk 2y ago
  > it was always a metric closely aligned with what customers want
  - liontwist 2y ago
    All OKRs help customers, but not all customer help meets an OKR. Finding and fixing product problems requires decentralized decision making and trust.
    - DrScientist 2y ago
      Indeed. What I find odd, is often the biggest proponents of the free market - with it's decentralised decision making process ( and redundancy of effort ), decide that total centralized, top down control is the best way to run their own company - as they think top down decisions and minimising waste through duplication is the way to go. As with all things - it's a balance of course.
      - zeroonetwothree 2y ago
        You might want to read "The Nature of the Firm", which discusses this.
        DrScientist 2y ago
        That's a slightly different angle - if I understand correctly it's about why firms exist at all - and if I were to summarise it's because lack of trust costs - a firms boundaries are created to balance transactional costs ( within the firm it's low, between firms or individuals it can dominate ) versus the costs of perhaps not being market efficient. My argument above is about not what shapes a firm's boundary but how it operates internally - too much top down control potentially risks exacerbating the risks associated with being a company and also potentially increases internal transactional costs as well - worse of both worlds! - all that ceremony around decision making, time spent justifying existences, inability to just act. Obviously as I said above - it's a balance - just as it is with country/international systems.
beAbU 2y ago
> a metric closely aligned with what customers want and which drives revenue for the company My experience says these two things are often mutually exclusive.
- hobs 2y ago
  Add a meta rule - only pursue revenue sources that make both of you money - that's called being a good company. Many people will say I am leaving stuff on the table, and I say that they are literally parasites.
  - PaulHoule 2y ago
    I was a Linux zealot in the 1990s. I had a conversation with a friend from college who was developing solutions based on Windows in Arizona who told me that he loved Microsoft because Windows was a platform that made people like him rich.
    - cpeterso 2y ago
      That matches Bill Gates’ definition of a platform: "A platform is when the economic value of everybody that uses it, exceeds the value of the company that creates it."
    - freedomben 2y ago
      Someday we'll get you back as a Linux zealot my friend :-)
stevenklein 2y ago
This is the crux of what's wrong with the original article IMO. Key results that are customer centric as opposed to "ship {thing}" help keep teams focused on the thing that actually matters. Of course there will be a tendency to try to game the metric, but the flip side of having customer centric goals is teams become feature factories, building idea after idea without constantly evaluating "are the things we're shipping driving the change in customer behavior they're intended to drive".
jayd16 2y ago
But this is such a good example of why these hard metrics are terrible. You actually probably don't want just 25% more, you probably want 30hz or 60hz or VRR support and you don't care about going from 92 to 118. The easy metric doesn't necessarily correlate to user desire. And like you mention, it disincentivizes reprioritizing with changing user desires simply because you didn't predict it or could come up with some other better metrics for something else. It's opposite of being agile but of course you see the same company claim they do both.
- michaelt 2y ago
  > You actually probably don't want just 25% more, you probably want 30hz or 60hz or VRR support and you don't care about going from 92 to 118. Then someone probably ought to tell the GPU review industry, because for the past 20+ years FPS on current popular games has been a key focus of their reporting :) I count 462 mentions of fps in https://www.tomshardware.com/reviews/gpu-hierarchy,4388.html https://www.tomshardware.com/reviews/gpu-hierarchy,4388.html
  - jayd16 2y ago
    You misunderstand. There are thresholds where FPS produces visible results and there is diminishing returns after 60fps as most displays cannot do more than that. When you make a game you'd want to hit those thresholds for as many users as possible. When you're reviewing cards all you can do is give a sampling of the landscape to get a rough sense of how the card would do in any given game.
- nine_k 2y ago
  > from 92 to 118 Or maybe yes; 120 Hz gaming monitors are a thing, and give some advantage. You probably want to target something like Doom Eternal, not BG3 though. As you see, it all depends on who is your customer, and then what matters to your customer. This is the closest proxy to the company's bottom line, and usually is not as fickle.
  - jayd16 2y ago
marcosdumay 2y ago
That's a product OKR, not a personal OKR. Yes, product OKRs can work, as long as you listen enough to feedback. In fact, you probably can't ever creating anything good without them. But I don't think most people even call them "OKR".

ricardobeat 2y ago

100% agree. There is a basic level of honesty required for this to work, which seems to have gone missing due to the "evolution" of corporate culture and the cash-rich environment which made actual productivity a secondary concern.

DanielHB 2y ago

In my job we have metrics, but they are not mandated to employees. There are no targets and stuff like that, they are just there if you want to know. Managers track it and try to optimize for it, but nobody bonuses or employment is on the line over them. Maybe some sales people have commission, but most of the customer acquisition is organic anyway. Funnily enough we have a weird situation right now, where we want to optimize the _cheaper_ plan of our product, meaning moving people from the more expensive plans to the cheaper one. It is a weird dynamic, but due to licensing our cheaper plan is actually far more profitable than the more expensive plans. Most clients wouldn't really miss anything from the more expensive plan. It is one of those rare situations where convincing people to switch to the cheaper plan aligns with what is best for the consumer.

ozim 2y ago
Well that sounds like good idea. As a manager you should have metrics because then you know if you tweak the process correctly. Not sharing it with employees and definitely not making any bonuses or pay increases depending on metrics. Because then employees will game your metrics and they will be useless.

aflukasz 2y ago

> For example, if you're designing GPUs and/or GPU drivers? If your next-generation product has the aim of providing 25% more frames per second in "Baldur's Gate 3" and "Call of Duty 6" while maintaining the same quality - that would be a good objective for the team, as it's closely aligned with what your customers want. I can get selection of particular gaming titles, but how do you come up with 25% goal? How is this closely aligned? Your users tell you they want ~30% gains? This seems to be completely ignoring a constant feedback loop between general aspirations for the product, operated timeframes, and conclusions from ongoing engineering R&D.

exe34 2y ago
I've never understood that sort of target - managers at McDonald's had them when I flipped burgers through uni, and they would just pull some numbers out of somewhere and get very excited about them. then they would nag customers until they hit those numbers, presumably.
michaelt 2y ago
> how do you come up with 25% goal? In organisations that do this kind of work, the 'marketing' department isn't just about placing adverts and writing blog posts. They also have people whose job is to keep track of what's going on in the market, and to estimate what your product needs in order to be competitive when it comes out. To take a simple example, the marketing team might have visited https://www.tomshardware.com/reviews/amd-radeon-rx-7900-xtx-and-xt-review-shooting-for-the-top/4 https://www.tomshardware.com/reviews/amd-radeon-rx-7900-xtx-... and found on 4K ultra settings that the "RX 7900 XTX" shows up as 90.3 fps average and the "RTX 4090" shows up as 112.6 fps average. And 112.6÷90.3 = 1.246 so if AMD want their next-gen flagship to outperform nvidia's current-gen flagship, they need 24.6% more FPS. Of course it's a lot more complicated than that in practice. They'd also consider value-for-money, non-flagship cards, whether users' monitors even support >120fps, guessing at what nvidia's next flagship will offer, interviewing big buyers like Dell, and so on. But that's the general gist of it.

liontwist 2y ago

And then customers complain that the driver is frequently crashing and the team is well aware but won’t fix the ticket because it doesn’t help them meet OKRs. What you really want is: - every team working to improve the customer experience - guided by leadership priorities and initiatives

madeofpalk 2y ago
> it was always a metric closely aligned with what customers want
- liontwist 2y ago
  All OKRs help customers, but not all customer help meets an OKR. Finding and fixing product problems requires decentralized decision making and trust.
  - DrScientist 2y ago
    Indeed. What I find odd, is often the biggest proponents of the free market - with it's decentralised decision making process ( and redundancy of effort ), decide that total centralized, top down control is the best way to run their own company - as they think top down decisions and minimising waste through duplication is the way to go. As with all things - it's a balance of course.
    - zeroonetwothree 2y ago
      You might want to read "The Nature of the Firm", which discusses this.
      - DrScientist 2y ago
        That's a slightly different angle - if I understand correctly it's about why firms exist at all - and if I were to summarise it's because lack of trust costs - a firms boundaries are created to balance transactional costs ( within the firm it's low, between firms or individuals it can dominate ) versus the costs of perhaps not being market efficient. My argument above is about not what shapes a firm's boundary but how it operates internally - too much top down control potentially risks exacerbating the risks associated with being a company and also potentially increases internal transactional costs as well - worse of both worlds! - all that ceremony around decision making, time spent justifying existences, inability to just act. Obviously as I said above - it's a balance - just as it is with country/international systems.

beAbU 2y ago

> a metric closely aligned with what customers want and which drives revenue for the company My experience says these two things are often mutually exclusive.

hobs 2y ago
Add a meta rule - only pursue revenue sources that make both of you money - that's called being a good company. Many people will say I am leaving stuff on the table, and I say that they are literally parasites.
- PaulHoule 2y ago
  I was a Linux zealot in the 1990s. I had a conversation with a friend from college who was developing solutions based on Windows in Arizona who told me that he loved Microsoft because Windows was a platform that made people like him rich.
  - cpeterso 2y ago
    That matches Bill Gates’ definition of a platform: "A platform is when the economic value of everybody that uses it, exceeds the value of the company that creates it."
  - freedomben 2y ago
    Someday we'll get you back as a Linux zealot my friend :-)

stevenklein 2y ago

This is the crux of what's wrong with the original article IMO. Key results that are customer centric as opposed to "ship {thing}" help keep teams focused on the thing that actually matters. Of course there will be a tendency to try to game the metric, but the flip side of having customer centric goals is teams become feature factories, building idea after idea without constantly evaluating "are the things we're shipping driving the change in customer behavior they're intended to drive".

jayd16 2y ago

But this is such a good example of why these hard metrics are terrible. You actually probably don't want just 25% more, you probably want 30hz or 60hz or VRR support and you don't care about going from 92 to 118. The easy metric doesn't necessarily correlate to user desire. And like you mention, it disincentivizes reprioritizing with changing user desires simply because you didn't predict it or could come up with some other better metrics for something else. It's opposite of being agile but of course you see the same company claim they do both.

michaelt 2y ago
> You actually probably don't want just 25% more, you probably want 30hz or 60hz or VRR support and you don't care about going from 92 to 118. Then someone probably ought to tell the GPU review industry, because for the past 20+ years FPS on current popular games has been a key focus of their reporting :) I count 462 mentions of fps in https://www.tomshardware.com/reviews/gpu-hierarchy,4388.html https://www.tomshardware.com/reviews/gpu-hierarchy,4388.html
- jayd16 2y ago
  You misunderstand. There are thresholds where FPS produces visible results and there is diminishing returns after 60fps as most displays cannot do more than that. When you make a game you'd want to hit those thresholds for as many users as possible. When you're reviewing cards all you can do is give a sampling of the landscape to get a rough sense of how the card would do in any given game.
nine_k 2y ago
> from 92 to 118 Or maybe yes; 120 Hz gaming monitors are a thing, and give some advantage. You probably want to target something like Doom Eternal, not BG3 though. As you see, it all depends on who is your customer, and then what matters to your customer. This is the closest proxy to the company's bottom line, and usually is not as fickle.
- jayd16 2y ago

marcosdumay 2y ago

That's a product OKR, not a personal OKR. Yes, product OKRs can work, as long as you listen enough to feedback. In fact, you probably can't ever creating anything good without them. But I don't think most people even call them "OKR".