4 ms·
I don't understands what the fallacy is. It is that an overreliance on data can overlook factors not in the data? Why is that a surprise. You would also have t
by paulpauper 5y ago
I don't understands what the fallacy is. It is that an overreliance on data can overlook factors not in the data? Why is that a surprise. You would also have to show that not relying on data would generate better results.
- warning26 5y ago> It is that an overreliance on data can overlook factors not in the data? Why is that a surprise? The vast majority of PMs I've worked with don't seem to understand this at all. If it's not in the metrics, it doesn't exist!
- 0xdeadbeefbabe 5y agoIf understanding is what you seek then https://www.youtube.com/watch?v=eNDsd798HR4 https://www.youtube.com/watch?v=eNDsd798HR4 is a nice long discussion on McNamara overlooking factors not in the data. The fallacy didn't earn its name for simple reasons.
- _Nat_ 5y agoIt's probably one of those things that might seem more obvious in simple cases, but might surprise folks in more complex cases. For example, a simple case: Say you go on vacation for a few weeks with a certain amount of cash to spend. Upon arriving at your destination, you immediately purchase some indulgence, and, hey, you're feeling better! Why not immediately keep spending as much as possible to maximize? For example, a complex case: Say you're leading a country. You set up policies that, after a few years, seem to have led to a higher GDP than was previously expected. Does that suggest that the policies are leading to a better future? In the simple cases, like the vacation-example above, it's easy enough to understand the scenario and what's going on. And we can imagine that, hey, immediately spending all of the cash might lead to poor consequences for the rest of the vacation, even if they display some quick-satisfaction at first. But in more complex cases, like with a country's policies leading to a higher GDP, stuff can get trickier. We might say that it's the lack of a top-level model: unlike in the vacation scenario, where we were easily able to predict that there'd be a lack of money later in the vacation, it might be harder to say what else might be going on besides the GDP going up. And all other things held equal, presumably a higher GDP would be better than a lower GDP, and therefore all evidence points toward the policies being a good idea, right?
- enkid 5y agoGDP is actually a great example. It's actually very easy to increase a country's GDP if that's the only thing you care about. You just borrow more money and then spend it. GDP is literally a measure of money changing hands inside a country. If you borrow as much money as you can and then spend it on things like infrastructure projects, you can instantly increase the GDP. Banks see the growing GDP figure and assume that its a good thing and let you borrow more money. That is, until they don't. This happened with Brazil in the 1970's, when the oil crisis made Brazil believe that they were going to face a downturn. To overcome this, they borrowed money and went on an infrastructure spending spree. This made Brazil look like an economic miracle, growing when everyone else was struggling. This encouraged more banks to lend to Brazil. The problem is the money was spent on short term growth instead of things that would more systematically grow the economy over the long term. The government (and lending banks) was substituting year-to-year GDP numbers for economic health. Once credit tightened in the early 1980's, Brazil's growth plummeted. This is of course an over simplification, but I think we are seeing similar problems in modern economies. People often cite China's GDP growth, but they don't balance that out with the debt they are taking on in order to finance that growth.
- torginus 5y agoIt probably tries to make the point that when a good metric becomes a target, it ceases to be a good metric - either through people gaming the metric, or because of diminishing returns. But I admit the article gets to its point in a muddy and circuitous way, typical of business school anecdotes.
- monkeybutton 5y agoI think the fallacy is when factors not captured or tracked by data are treated as non-existing or consequential. In Vietnam, they tried to quantify how pacified each village was as measurement of success. It did not work. This isn't saying no data is better, just that data on its own does not win a war, or the hearts and minds of villagers. See also: https://www.theatlantic.com/technology/archive/2017/10/the-computer-that-predicted-the-us-would-win-the-vietnam-war/542046/ https://www.theatlantic.com/technology/archive/2017/10/the-c...
- potatolicious 5y agoThere are a bunch of factors and implications from the piece, which I had hoped they'd go into. But some of the more obvious implications which are often more organizational than they have to do with the data per se: - the data you are collecting may not include factors that are consequential to outcomes. Treating these unmeasured factors as inconsequential is hazardous. The piece specifically mentions this effect. This effect is not a surprise but yet many organizations fall into this trap so it seems to be worth mentioning. - the difficulty of measuring some factors will result in their exclusion from the dataset. Some things are intrinsically hard to measure, and many organizations will as a result refuse to measure them, and make decisions without them. There needs to be a conscious and active process organizationally to resist this and find effective ways of measuring them. - the difficulty of measuring some factors will result in easier but less useful proxies being used in their place. Same as above just with a somewhat different outcome. Organizations are often blind to this happening as they come to believe the proxy is as good as the real measure (or sometimes even that the proxy is the measure). A good industry example of this is clickthrough rates being treated synonymously with audience interest or content quality. And a point I wish the piece made but did not: - measuring something does not grant automatic understanding of the phenomenon and gives you no predictive power. It's one thing to quantifiably know the blue button gets more clicks than the green button, but that grants you no insight into why. Many tech companies fall into this trap - where despite investing heavily in experimentation their modeling of the product space doesn't improve over time, since they fail to take the step to convert observation to generalizable hypotheses that improve their model for the product and market. This last effect IMO is huge in our industry, and is why the same low-level experiments are being re-done over and over again. This creates more product churn and reduces your product velocity - since the lack of proven product models means you're mostly flying blind and using ex-post-facto experimentation on live users to figure out what to do. Experimenting on button colors is all well and good, but the end result of that data should be a color theory that explains what colors to use when, not forever A/B testing every single button color for the rest of time.
- facorreia 5y agoMy understanding is that the fallacy is cherry-picking a few data points that are easy to measure and then claiming that these are the most important data points, and then optimizing execution for moving them in the "right" direction. This is described in the metaphor of "searching under the streetlight". The key to this con is to act confident and to challenge any objections as lacking supporting data.
- asveikau 5y ago> Why is that a surprise It's surprising to many. Many people will decide they are "data driven" and act like this gives them infallibility and total correctness. Lots of people need to be told this.
- mcguire 5y agoThe real question isn't "Why is that a surprise?"; it's "Why does it keep happening?"