4 ms·

It’s a great example of how people will use an easy quantification just because it exists. There’s a popular mindset that seems to think you can’t argue against

It’s a great example of how people will use an easy quantification just because it exists. There’s a popular mindset that seems to think you can’t argue against numbers and math because they’re cold and scientific. Who cares if its a stupid number, it’s mathematically/scientifically correct!

keiferski 6y ago
New tools, same old tricks. We've just upgraded from turtle shells to web apps.
jimmaswell 6y ago
A measure that becomes a target often stops being a good measure.
- jjoonathan 6y ago
  Yes, but sometimes it was garbage in the first place, too.
- ketzo 6y ago
  Huh. This is actually quite insightful to me. I wonder if this applies more broadly? What from your own life makes you say that?
  - InitialLastName 6y ago
    Not GP, but that's a formation of Goodhart's law [0], which comes up in a lot of situations. Notable potential Goodhart's law situations: - Testing in education - GDP (or a number of other economic indicators) as a measure of economic performance/target for economic performance - Healthcare outcomes along a number of axes [0] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
  - banana_giraffe 6y ago
    It's a variant of Goodhart's law. Can't speak for the person you're replying to, but absent good management, I've seen review goals turn into nonsense. "You had a goal to fix 5 bugs this quarter, and only fixed 4", "but one was a massive bug that would have cost us $x!", "but you didn't fix 5 bugs, so no bonus" You see it happen on different levels a surprising number of times.
    - indigochill 6y ago
      Anecdotally, when I worked in customer support, we were explicitly told that we had to hit a certain number of replies to customers in our ticket system. So I played by the rules: I very clearly grabbed all the easy-to-reply tickets, sent the easy reply, and dropped them if they got hard. Appropriately, a manager spoke to me telling me this was unacceptable (though he did concede the goal was stupid), and gratifyingly the official goals soon changed to reflect an appropriate balance of consistency with a customer as well as some degree of overall throughput. The drawback there is that accurately reflecting what they want in hard metrics might not really be possible because it does come down to a judgment call where the exact balance between support quality and throughput lies (and it's often a shifting balance as well, depending how much support load there is). Which is probably why managers fall back on stupid but easy-to-measure metrics.
  - deleted 6y ago
    [deleted]
  - dkn775 6y ago
    It does in my experience, I work for a gov public safety agency and the numbers they show you are one part of the story. In public policy we learned how you determine what the thresholds are (When does good become excellent?), as well as things like what does improvement mean (a ten percent decrease in raw number? ten percent decrease from last years average?) mean that the framing of metrics must always be examined. Not to mention the litany of inside baseball stuff regarding how various things like 911 calls, car crashes, crimes, and population figures are captured, analyzed, and understood. Outside of numbers. The same logic applies to fact checking sites which is why I always question why so many people use them to refute arguements. The authors decided what was deemed a good source. A dirty secret I’ve noticed of the most controversial issues at the end of the day can be argued successfully both ways, so the fact checkers effectively choose the side they agree with. For example, should Texas be demonized for removing their mask mandates? Lot of areas in Texas are very remote. Texas is more insular of a state than say New Jersey. Federalism and US legal history typically affords states the latitude to make decisions for themselves based on their populations desires. To what extent does interstate commerce (flying out of Texas for example) justify outside/Federal force to upend Texas sovereignty? Lot of people try to argue Texas is just wrong but it’s deeper than that.
africanboy 6y ago
that's not how it works though... you can't argue against numbers, you can absolutely argue against arbitrary numbers (like in this example the premise is that the app produces reliable results, which it does not and anyway it should be proved with good, hard to dispute, numbers)
- Barrin92 6y ago
  OP's point is that all numbers are arbitrary (to varying degrees). The act of quantifying always imposes some sort of framework on what you're trying to look at, and we live in an age of pretty vulgar scientism where this is associated with 'objectivity'. From 'personality scores' for applicants, to social credit scores, and in this case the quality of writing in a newspaper.
  - 6gvONxR4sf7o 6y ago
    Right. The number 3 is inarguable. That 4+2=6 is inarguable. The applicability of a give statistic to a given situation or argument is a much more difficult proposition, but people too frequently treat a statistic as relevant just because it's easy to compute. After all mercury is in retrograde, and that's a scientific fact.
  - africanboy 6y ago
    and they are all non arbitrary (to varying degrees), but arguably some are objectively more arbitrary than others.

keiferski 6y ago

New tools, same old tricks. We've just upgraded from turtle shells to web apps.

jimmaswell 6y ago

A measure that becomes a target often stops being a good measure.

jjoonathan 6y ago
Yes, but sometimes it was garbage in the first place, too.
ketzo 6y ago
Huh. This is actually quite insightful to me. I wonder if this applies more broadly? What from your own life makes you say that?
- InitialLastName 6y ago
  Not GP, but that's a formation of Goodhart's law [0], which comes up in a lot of situations. Notable potential Goodhart's law situations: - Testing in education - GDP (or a number of other economic indicators) as a measure of economic performance/target for economic performance - Healthcare outcomes along a number of axes [0] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- banana_giraffe 6y ago
  It's a variant of Goodhart's law. Can't speak for the person you're replying to, but absent good management, I've seen review goals turn into nonsense. "You had a goal to fix 5 bugs this quarter, and only fixed 4", "but one was a massive bug that would have cost us $x!", "but you didn't fix 5 bugs, so no bonus" You see it happen on different levels a surprising number of times.
  - indigochill 6y ago
    Anecdotally, when I worked in customer support, we were explicitly told that we had to hit a certain number of replies to customers in our ticket system. So I played by the rules: I very clearly grabbed all the easy-to-reply tickets, sent the easy reply, and dropped them if they got hard. Appropriately, a manager spoke to me telling me this was unacceptable (though he did concede the goal was stupid), and gratifyingly the official goals soon changed to reflect an appropriate balance of consistency with a customer as well as some degree of overall throughput. The drawback there is that accurately reflecting what they want in hard metrics might not really be possible because it does come down to a judgment call where the exact balance between support quality and throughput lies (and it's often a shifting balance as well, depending how much support load there is). Which is probably why managers fall back on stupid but easy-to-measure metrics.
- deleted 6y ago
  [deleted]
- dkn775 6y ago
  It does in my experience, I work for a gov public safety agency and the numbers they show you are one part of the story. In public policy we learned how you determine what the thresholds are (When does good become excellent?), as well as things like what does improvement mean (a ten percent decrease in raw number? ten percent decrease from last years average?) mean that the framing of metrics must always be examined. Not to mention the litany of inside baseball stuff regarding how various things like 911 calls, car crashes, crimes, and population figures are captured, analyzed, and understood. Outside of numbers. The same logic applies to fact checking sites which is why I always question why so many people use them to refute arguements. The authors decided what was deemed a good source. A dirty secret I’ve noticed of the most controversial issues at the end of the day can be argued successfully both ways, so the fact checkers effectively choose the side they agree with. For example, should Texas be demonized for removing their mask mandates? Lot of areas in Texas are very remote. Texas is more insular of a state than say New Jersey. Federalism and US legal history typically affords states the latitude to make decisions for themselves based on their populations desires. To what extent does interstate commerce (flying out of Texas for example) justify outside/Federal force to upend Texas sovereignty? Lot of people try to argue Texas is just wrong but it’s deeper than that.

africanboy 6y ago

that's not how it works though... you can't argue against numbers, you can absolutely argue against arbitrary numbers (like in this example the premise is that the app produces reliable results, which it does not and anyway it should be proved with good, hard to dispute, numbers)

Barrin92 6y ago
OP's point is that all numbers are arbitrary (to varying degrees). The act of quantifying always imposes some sort of framework on what you're trying to look at, and we live in an age of pretty vulgar scientism where this is associated with 'objectivity'. From 'personality scores' for applicants, to social credit scores, and in this case the quality of writing in a newspaper.
- 6gvONxR4sf7o 6y ago
  Right. The number 3 is inarguable. That 4+2=6 is inarguable. The applicability of a give statistic to a given situation or argument is a much more difficult proposition, but people too frequently treat a statistic as relevant just because it's easy to compute. After all mercury is in retrograde, and that's a scientific fact.
- africanboy 6y ago
  and they are all non arbitrary (to varying degrees), but arguably some are objectively more arbitrary than others.