4 ms·

I am not "Roger," but I can recognize someone who has long and practical experience with managing metrics and KPIs and their interaction with process improvemen

I am not "Roger," but I can recognize someone who has long and practical experience with managing metrics and KPIs and their interaction with process improvement. Instead of an "expiration date" I would encourage you to define a "re-evaluation date" that allows enough time to judge the impact and efficacy of the metrics proposed and make course corrections as needed (each with its own review dates). One good book on the positive impact of a metric that everyone on a team or organization understands is "The Great Game of Business" by Jack Stack https://www.amazon.com/Great-Game-Business-Expanded-Updated-ebook/dp/B00B3GMH4W/ https://www.amazon.com/Great-Game-Business-Expanded-Updated-... I reviewed it at https://www.skmurphy.com/blog/2010/03/19/the-business-is-everyones-business/ https://www.skmurphy.com/blog/2010/03/19/the-business-is-eve... Here is a quote to give you a flavor of his philosophy: "A business should be run like an aquarium, where everybody can see what's going on--what's going in, what's moving around, what's coming out. That's the only way to make sure people understand what you're doing, and why, and have some input into deciding where you are going. Then, when the unexpected happens, they know how to react and react quickly. " Jack Stack in "Great Game of Business."

shadowsun7 2y ago
I should note that this essay kicks off an entire series that eventually culminates in a detailed examination of the Amazon Weekly Business Review (which takes some time to get to because of a) an NDA, and b) it took some time to test it in practice). The Goodhart’s Law essay uses publicly available information about the WBR to explain how to defeat Goodhart’s Law (since the ideas it draws from are five decades old); the WBR itself is a two decades-old mechanism on how to actually accomplish these high-falutin’ goals. https://commoncog.com/the-amazon-weekly-business-review/ https://commoncog.com/the-amazon-weekly-business-review/ Over the past year, Roger and I have been talking about the difficulty of spreading these ideas. The WBR works, but as the essay shows, it is an interlocking set of processes that solves for a bunch of socio-technical problems. It is not easy to get companies to adopt such large changes. As a companion to the essay, here is a sequence of cases about companies putting these ideas to practice: https://commoncog.com/c/concepts/data-driven/ https://commoncog.com/c/concepts/data-driven/ The common thing in all these essays is that it doesn’t stop at high-falutin’ (or conceptual) recommendation, but actually dives into real world application and practice. Yes, it’s nice to say “let’s have a re-evaluation date.” But what does it actually look like to get folks to do that at scale? Well, the WBR is one way that works in practice, at scale, and with some success in multiple companies. And we keep finding nuances in our own practice: https://x.com/ejames_c/status/1849648179337371816 https://x.com/ejames_c/status/1849648179337371816
- skmurphy 2y ago
  It looks like any other decision record where you set a date to evaluate the impact of a policy or course of action and make sure it's working out the way that you had anticipated.
  - shadowsun7 2y ago
    And how are you going to tell that when you have a) variation (that is, every metric wiggles wildly)? And also b) how are you able to tell if it has or hasn’t impacted other parts of your business if you do not have a method for uncovering the causal model of your business (like that aquarium quote you cited earlier?) Reality has a lot of detail. It’s nice to quote books about goals. It’s a different thing entirely to achieve them in practice with a real business.
    - skmurphy 2y ago
      I agree that reality is complex, but I worry you are conflating the challenges of running an Amazon-scale business with running the smaller businesses that most of the entrepreneurs on HN will need to manage. I thought Roger offered a more practical approach in about 10% of the words that you took. I am sorry if I have offended you; I was trying to save the entrepreneurs on HN time. As to Jack Stack's book, I think the genius of his approach is communicating simple decision rules to the folks on the front line instead of trying to establish a complex model at the executive level that can become more removed from day-to-day realities. In my experience, which involves working in a variety of roles in startups and multi-billion dollar businesses over the better part of five decades, simple rules updated based on your best judgment risk "extinction by instinct" but outperform the "analysis paralysis" that comes from trying to develop overly complex models. Reasonable men may differ.
      - shadowsun7 2y ago
        This comment is for HN readers who might be interested in solutions. My two questions (a) and (b) were not rhetorical. Let’s get concrete. a) You are advising a company to “check back after a certain period”. After the certain period, they come back to you with the following graph: https://commoncog.com/content/images/2024/01/prospect_calls_change_2.f22a294d9d364d908902b5f40f3bd223.png https://commoncog.com/content/images/2024/01/prospect_calls_... “How did we do? Did we improve?” How do you answer? Notice that this is a problem regardless of whether you are a big company or a small company. b) 3 months later, your client comes back and asks: “we are having trouble with customer support. How do we know that it’s not related to this change we made?” With your superior experience working with hundreds of startups, you are able to tell them if it is or isn’t after some investigation. Your client asks you: “how can we do that for ourselves without calling on you every time we see something weird?” How do you answer? (My answers are in the WBR essay and the essay that comes immediately before that, natch) It is a common excuse to wave away these ideas with “oh, these are big company solutions, not applicable to small businesses.” But a) I have applied these ideas to my own small business and doubled revenue; also b) in 1992 Donald Wheeler applied these methods to a small Japanese night club and then wrote a whole book about the results: https://www.amazon.sg/Spc-Esquire-Club-Donald-Wheeler/dp/0945320302 https://www.amazon.sg/Spc-Esquire-Club-Donald-Wheeler/dp/094... Wheeler wanted to prove, (and I wanted to verify), that ‘tools to understand how your business ACTUALLY works’ are uniformly applicable regardless of company size. If anyone reading this is interested in being able to answer confidently to both questions, I recommend reading my essays to start with (there’s enough in front of the paywall to be useful) and then jump straight to Wheeler. I recommend Understanding Variation, which was originally developed as a 1993 presentation to managers at DuPont (which means it is light on statistics).
krisstring 2y ago
While Goodhart’s Law often occurs because of a narrow focus on a metric without understanding its role in the larger system, the approach in Jack Stack’s The Great Game of Business is to make targets an educational tool, teaching employees how to interpret and impact those targets responsibly. GGOB, by 1. involving employees in decision-making and teaching them about metrics, 2. giving them a line-of-sight for how their contribution impacts the overall business, and 3. providing a stake in the outcome creates collective accountability and success, and reduces the likelihood of metric manipulation. Bottom line: GGOB recognizes that business success takes everyone, at all levels, and values the input of each employee, right down to the part-time janitor. The metrics are used as tools, like the scoreboard in baseball, to guide decision making and establish what winning as a team looks like. It all comes down to education and getting everyone aligned and pulling in the same direction.
- skmurphy 2y ago
  I agree it's really about effective delegation, it acknowledges the risks that Goodhart warns about and suggests how to manage if not avoid them.

shadowsun7 2y ago

I should note that this essay kicks off an entire series that eventually culminates in a detailed examination of the Amazon Weekly Business Review (which takes some time to get to because of a) an NDA, and b) it took some time to test it in practice). The Goodhart’s Law essay uses publicly available information about the WBR to explain how to defeat Goodhart’s Law (since the ideas it draws from are five decades old); the WBR itself is a two decades-old mechanism on how to actually accomplish these high-falutin’ goals. https://commoncog.com/the-amazon-weekly-business-review/ https://commoncog.com/the-amazon-weekly-business-review/ Over the past year, Roger and I have been talking about the difficulty of spreading these ideas. The WBR works, but as the essay shows, it is an interlocking set of processes that solves for a bunch of socio-technical problems. It is not easy to get companies to adopt such large changes. As a companion to the essay, here is a sequence of cases about companies putting these ideas to practice: https://commoncog.com/c/concepts/data-driven/ https://commoncog.com/c/concepts/data-driven/ The common thing in all these essays is that it doesn’t stop at high-falutin’ (or conceptual) recommendation, but actually dives into real world application and practice. Yes, it’s nice to say “let’s have a re-evaluation date.” But what does it actually look like to get folks to do that at scale? Well, the WBR is one way that works in practice, at scale, and with some success in multiple companies. And we keep finding nuances in our own practice: https://x.com/ejames_c/status/1849648179337371816 https://x.com/ejames_c/status/1849648179337371816

skmurphy 2y ago
It looks like any other decision record where you set a date to evaluate the impact of a policy or course of action and make sure it's working out the way that you had anticipated.
- shadowsun7 2y ago
  And how are you going to tell that when you have a) variation (that is, every metric wiggles wildly)? And also b) how are you able to tell if it has or hasn’t impacted other parts of your business if you do not have a method for uncovering the causal model of your business (like that aquarium quote you cited earlier?) Reality has a lot of detail. It’s nice to quote books about goals. It’s a different thing entirely to achieve them in practice with a real business.
  - skmurphy 2y ago
    I agree that reality is complex, but I worry you are conflating the challenges of running an Amazon-scale business with running the smaller businesses that most of the entrepreneurs on HN will need to manage. I thought Roger offered a more practical approach in about 10% of the words that you took. I am sorry if I have offended you; I was trying to save the entrepreneurs on HN time. As to Jack Stack's book, I think the genius of his approach is communicating simple decision rules to the folks on the front line instead of trying to establish a complex model at the executive level that can become more removed from day-to-day realities. In my experience, which involves working in a variety of roles in startups and multi-billion dollar businesses over the better part of five decades, simple rules updated based on your best judgment risk "extinction by instinct" but outperform the "analysis paralysis" that comes from trying to develop overly complex models. Reasonable men may differ.
    - shadowsun7 2y ago
      This comment is for HN readers who might be interested in solutions. My two questions (a) and (b) were not rhetorical. Let’s get concrete. a) You are advising a company to “check back after a certain period”. After the certain period, they come back to you with the following graph: https://commoncog.com/content/images/2024/01/prospect_calls_change_2.f22a294d9d364d908902b5f40f3bd223.png https://commoncog.com/content/images/2024/01/prospect_calls_... “How did we do? Did we improve?” How do you answer? Notice that this is a problem regardless of whether you are a big company or a small company. b) 3 months later, your client comes back and asks: “we are having trouble with customer support. How do we know that it’s not related to this change we made?” With your superior experience working with hundreds of startups, you are able to tell them if it is or isn’t after some investigation. Your client asks you: “how can we do that for ourselves without calling on you every time we see something weird?” How do you answer? (My answers are in the WBR essay and the essay that comes immediately before that, natch) It is a common excuse to wave away these ideas with “oh, these are big company solutions, not applicable to small businesses.” But a) I have applied these ideas to my own small business and doubled revenue; also b) in 1992 Donald Wheeler applied these methods to a small Japanese night club and then wrote a whole book about the results: https://www.amazon.sg/Spc-Esquire-Club-Donald-Wheeler/dp/0945320302 https://www.amazon.sg/Spc-Esquire-Club-Donald-Wheeler/dp/094... Wheeler wanted to prove, (and I wanted to verify), that ‘tools to understand how your business ACTUALLY works’ are uniformly applicable regardless of company size. If anyone reading this is interested in being able to answer confidently to both questions, I recommend reading my essays to start with (there’s enough in front of the paywall to be useful) and then jump straight to Wheeler. I recommend Understanding Variation, which was originally developed as a 1993 presentation to managers at DuPont (which means it is light on statistics).

krisstring 2y ago

While Goodhart’s Law often occurs because of a narrow focus on a metric without understanding its role in the larger system, the approach in Jack Stack’s The Great Game of Business is to make targets an educational tool, teaching employees how to interpret and impact those targets responsibly. GGOB, by 1. involving employees in decision-making and teaching them about metrics, 2. giving them a line-of-sight for how their contribution impacts the overall business, and 3. providing a stake in the outcome creates collective accountability and success, and reduces the likelihood of metric manipulation. Bottom line: GGOB recognizes that business success takes everyone, at all levels, and values the input of each employee, right down to the part-time janitor. The metrics are used as tools, like the scoreboard in baseball, to guide decision making and establish what winning as a team looks like. It all comes down to education and getting everyone aligned and pulling in the same direction.

skmurphy 2y ago
I agree it's really about effective delegation, it acknowledges the risks that Goodhart warns about and suggests how to manage if not avoid them.