21 ms·

The McNamara fallacy: Measurement is not understanding

365 points by wenc 5y ago

deleted 5y ago
[deleted]
aeternum 5y ago
From a game-theory POV, all wars are caused by misunderstandings / partial information. If each side really knew the true strength of the other side, it would be clear which would win and therefore actual war is illogical and unnecessary.
- hackeraccount 5y ago
  Is that true? What if the situation is that I am currently stronger then you but can expect to become weaker in the future? Why not force a war now if I can expect to win? I still haven't finished it (too depressing) but my reading of the book The Sleepwalkers about the events leading up to WWI is that all the parties thought they were in that place; even if they all had a full picture it's certainly possible those beliefs would in fact be true.
  - aeternum 5y ago
    If I know that you are currently stronger and would ultimately win a war, I should logically concede now since I will lose anyway. If I know that you are currently stronger but we both also know that I can draw out a war such that I eventually become stronger, then you shouldn't go to war since you will ultimately lose.
    - vkou 5y ago
      > If I know that you are currently stronger and would ultimately win a war, I should logically concede now since I will lose anyway. You'll often lose less if you fight to a loss. In fact, you often won't have to fight at all if you make it clear to your adversary that you will fight to a loss. If I'm a street thug, I'm not going to try to mug someone who clearly communicates that their response to a mugging is detonating their suicide vest. I will, however, repeatedly victimize someone who clearly communicate that they are a pushover. This is an iterative game, against the same players. Loss aversion is not a winning strategy to pursue in the long-term. Something resembling tit-for-tat is.
- vlovich123 5y ago
  That's a fallacious line of reasoning on several fronts. A) The beneficiaries of a war need not be the ones who pay the cost. If you personally stand to benefit from the war itself, regardless of win or lose, you want the war. B) It's applying a rational economics approach when it's pretty clear that humans aren't rational. Japan kept fighting even when it was clear they would lose. Pride, image, your future after the war. All things that impact the decision that gets made. C) War isn't typically total like it was in WWII. Most wars are just skirmishes or prolonged skirmishes that don't meaningfully change borders. In such a scenario, a single "war" loss may still be an important fight to have in the middle of the overall conflict (i.e. to signal to your opponent that you're not a pushover and that continuing on their current path has consequences). D) Win/lose isn't the only outcome of war. Lose/lose is also another scenario. Additionally, you might lose the war but winning may not be the goal. For example, maybe it only takes me 1/10th the resources for me to wage the war that it takes for you to win. A single war might cost me 100 units, but it's important for me to be able to cause you to spend 1000 units because then I have a 900 unit advantage in some other area. There's many more, but I don't think imperfect information is the only reason war exists.
  - tomrod 5y ago
    (1) Externalities are handled in the game theory literature (2) Preference definitions are not in scope for most game theory lit; what you're describing are preferences and expected utility, which ARE in the literature and metamodel (3) This is not really true. It depends on how big you are and how big your coalition is. Consider that Saddam Hussain is no longer in charge of Iraq. (4) Yes, potential outcomes are considered under game theory. I get your point that OP's statement feels reductionist, but I wanted to explain that the OP is actually correct.
    - mcguire 5y ago
      You didn't work at Rand in the 1960s, did you? :-) Anyway, the OP may be correct, but is also completely useless since it is impossible to know your own true strength, much less that of the other side.
- 13415 5y ago
  Wait a minute. Shouldn't the goal of the defender be to make it clear to the attacker that they would incur losses that outweigh possible wins for the attacker, thereby making the attack a lose-lose scenario? This is only possible if the defender makes it credible to the potential attacker that they will defend their country even when the costs would be extremely high and they cannot possibly win the conflict. In other words, at least from deterrence point of view it's not about winning, it's about making sure the attacker would overall lose more than gain (which is not the same as winning against the attacker).
- crazy1van 5y ago
  > If each side really knew the true strength of the other side, it would be clear which would win This is too simplistic. Often wars are heavily influenced by things other than just the strengths of both sides. For example - What if the ground hadn't been soaked from days of rain at Agincourt and Waterloo? What if the Germans hadn't held back their armor reinforcements for so long on D-Day?
  - asdff 5y ago
    They'd still lose. Wars are fought not from individual battles but from logistics. Modern military's don't even engage if there isn't a lopsided power imbalance favoring their success.
    - 0xdeadbeefbabe 5y ago
      Yeah like in Black Hawk Down.
      - asdff 5y ago
        In real life the battle of mogadishu saw Americans with 10x fewer casualties. Most modern battles have numbers like this with an order of magnitude fewer casualties on the American side, because American logistic planners try to ensure a massive power advantage.
        mcguire 5y ago
        And yet...
        asdff 5y ago
        And yet what? Are you not convinced by the casualty data?
        mcguire 5y ago
        I don't seem to recall Mogadishu being any kind of "victory". It's almost like the casualty data didn't tell the whole story.
- s28l 5y ago
  There is a lot to disagree with here. First, the article itself provides an example of an army with a superior fighting force that had far fewer casualties than the other side, yet still lost the war. That was due to the civilians back at home lacking the stomach for a long, protracted war abroad. So even if one side has superior strength, and even if the side with the superior strength wins every battle, they might still lose the war. Another issue is your implicit assumption that the side with the superior fighting force will always win the war, but I don't think you can make that assumption. Just like the better football team can lose to the underdog, there is a stochastic element to warfare. A sudden bit of bad weather can turn the tide of a battle. There are countless other elements that are unobservable and unpredictable that can decide which side wins. There are also asymmetric payoffs to going to war. A nation might have a slim chance of victory, but the cost of defeat or surrender might be genocide or subjection. How do you assign a payoff to the choice "surrender" when the outcome is the destruction of your nation as it once existed? [0] https://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_and_mixed_strategies https://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_an...
- aeturnum 5y ago
  This seems untrue. There are also genuine uncertainties about the future: things that neither side knows but will impact the outcome. Because of that, there is always a range of outcomes for a conflict, and so both sides would still have an incentive to carry out the war. Consider, for instance, if there are two countries (A and B) looking at conflict with perfect information. Given their relative strengths, the war will cost both $1 billion, but A will win and reap $1 billion + $1 in plunder. In your model, B will always surrender and give A $1, but the reality is there is a big spread about both the costs and the gains. Lots depends on how well each side fights and responds, even with perfect knowledge of the other. So...you know, if it's the US versus...the Philippines[1], then sure, the uncertain range of outcomes is small enough that rational thing for the Philippines to do is surrender. But for countries that are even reasonably well-matched, they would want to fight with the goal of making the conflict unattractive for the other side. Probably the best historical example of this is the USSR in Afghanistan. [1] I am sure the US would never invade and occupy them just because the US found it convenient ;)
  - thaumasiotes 5y ago
    > Consider, for instance, if there are two countries (A and B) looking at conflict with perfect information. Given their relative strengths, the war will cost both $1 billion, but A will win and reap $1 billion + $1 in plunder. In your model, B will always surrender and give A $1 In the perfect information scenario, B will surrender more than $1. B is better off at any tribute level up to $1,000,000,000.
    - aeturnum 5y ago
      Sure, I didn't go far into detail, but it's actually worse than either of us has said because once B surrenders anything ($1 or $1,000,000,000), they are at even greater disadvantage and "logically" should surrender more money. In fact, they should surrender the entire country, as once they are at a disadvantage A can keep coming back forever. Effectively, the idea that war would not happen with perfect information suggests that the logical thing to do is for all countries to surrender control to the strongest country immediately.
- chasd00 5y ago
  i'll add that there's a lot of people that see death as the door to paradise and fighting for their cause as the key to that door. No amount of logic or information is going to stop them. when you're dealing with the mentally ill, all bets are off.
- deleted 5y ago
  [deleted]
paulpauper 5y ago
I don't understands what the fallacy is. It is that an overreliance on data can overlook factors not in the data? Why is that a surprise. You would also have to show that not relying on data would generate better results.
- warning26 5y ago
  > It is that an overreliance on data can overlook factors not in the data? Why is that a surprise? The vast majority of PMs I've worked with don't seem to understand this at all. If it's not in the metrics, it doesn't exist!
- 0xdeadbeefbabe 5y ago
  If understanding is what you seek then https://www.youtube.com/watch?v=eNDsd798HR4 https://www.youtube.com/watch?v=eNDsd798HR4 is a nice long discussion on McNamara overlooking factors not in the data. The fallacy didn't earn its name for simple reasons.
- _Nat_ 5y ago
  It's probably one of those things that might seem more obvious in simple cases, but might surprise folks in more complex cases. For example, a simple case: Say you go on vacation for a few weeks with a certain amount of cash to spend. Upon arriving at your destination, you immediately purchase some indulgence, and, hey, you're feeling better! Why not immediately keep spending as much as possible to maximize? For example, a complex case: Say you're leading a country. You set up policies that, after a few years, seem to have led to a higher GDP than was previously expected. Does that suggest that the policies are leading to a better future? In the simple cases, like the vacation-example above, it's easy enough to understand the scenario and what's going on. And we can imagine that, hey, immediately spending all of the cash might lead to poor consequences for the rest of the vacation, even if they display some quick-satisfaction at first. But in more complex cases, like with a country's policies leading to a higher GDP, stuff can get trickier. We might say that it's the lack of a top-level model: unlike in the vacation scenario, where we were easily able to predict that there'd be a lack of money later in the vacation, it might be harder to say what else might be going on besides the GDP going up. And all other things held equal, presumably a higher GDP would be better than a lower GDP, and therefore all evidence points toward the policies being a good idea, right?
dempedempe 5y ago
This article is good, but not great - the author only gives one example of how quantitative-only reasoning can be bad (the example of the poppies). The other "example" is just the US military lying. There are also no specific examples of non-quantitative reasoning that, if ignored, would be damaging. I feel like the Wikipedia article does a better job explaining this: https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy Also, why is there an entire webpage dedicated to this?
- enkid 5y ago
  The example with the US military lying certainly demonstrates another issue with over valuing metrics - making sure you have good data. People who get wrapped around metrics also tend to not look at the data they are being presented. One of the issues McNamara had was he was receiving inflated body count figures. [0] Not only was he measuring the wrong thing, he was measuring it badly. [0] https://en.wikipedia.org/wiki/Vietnam_War_body_count_controversy https://en.wikipedia.org/wiki/Vietnam_War_body_count_controv...
  - the_af 5y ago
    It's also a good example of perverse incentives or "gaming the metrics" (in a horrifying way). If the main measure of success was body count, and every kill was considered an enemy by default, this encouraged just killing people and counting them as successes.
    - shellback3 5y ago
      I was in Vietnam in a Naval Support Group. I had conversations with army officers that had combat experience and learned that since no statistics were collected about dead civilians all the dead were counted as enemy soldiers - and everyone from the top down knew it.
- hummusandsushi 5y ago
  The webpage appears to be made to serve as a warning to data-driven businesses to not fall into the same perverse set of incentives that McNamara created, to encourage business managers to diversify their accounts of the success of their business beyond just the quantitative narrative.
snidane 5y ago
I wonder if it gets widespread to oppose the dominant hyperquantitative, every fart tracked in jira, can't measure - can't manage trend. Which was never pushed by anybody from the industry, but got spread by project management frauds. https://deming.org/myth-if-you-cant-measure-it-you-cant-manage-it/ https://deming.org/myth-if-you-cant-measure-it-you-cant-mana...
eumoria 5y ago
McNamara basically admits this himself in the documentary The Fog of War: https://en.wikipedia.org/wiki/The_Fog_of_War https://en.wikipedia.org/wiki/The_Fog_of_War It's worth a watch but it's very soft on him and his role. Still a very good documentary.
- ProAm 5y ago
  Great documentary. I felt it was a person trying to come clean and ease his conscious on his death bed.
  - 2OEH8eoCRo0 5y ago
    Agreed. Great documentary. The part where Castro said he urged Russia to launch their nukes from Cuba to the US knowing it would destroy Cuba was chilling. Humans are not always logical. Don't assume somebody wont drag an entire country or the world to total destruction for some deranged cause. I didn't get the death bed vibe from McNamara but I definitely felt that he was genuinely reflecting on the past. The documentary on Rumsfeld was the polar opposite. I could also see Rumsfeld not wanting to give the enemy of an ongoing conflict any shred of material. It makes for a less interesting documentary.
    - nickdothutton 5y ago
      Morris himself said something like he didnt really feel he got to know Rumsfeld and had no real idea what was in his mind compared to his previous interview with McNamara.
      - wayoutthere 5y ago
        Guess Donald himself was one of the “unknown unknowns”
        codechad 5y ago
jscode 5y ago
Quick story: I was the CFO for a company that sold to a private equity group (PEG). I took over as the CEO as the founders retired, leaving me to deal with the PEG. It quickly became apparent that the PEG managers looked at everything through the lens of an Excel spreadsheet. These guys were brilliant attorneys and analysts but lacked experience building businesses and managing teams. Ultimately, they couldn’t add much value in terms of operations or strategy, but they were great at financial modeling/quantitative analysis and forcing us to justify expenses. That may sound good at first—eliminating wasteful spending—but it ultimately led to the gradual erosion of the company culture and employee loyalty. It’s easy to cut benefits and pay given that many workers lack the leverage to do anything about it, while it’s much harder to reduce hard costs like materials and equipment. That meant employees just kept getting squeezed, and it was surprisingly difficult to quantify the impact that terminating an employee or cutting benefits would have on morale/culture/performance. The moral of the story is that people with analyst mindsets play an essential role in our economy, but sometimes giving those people power over large organizations can have disastrous consequences. There truly is a disconnect between measurement and understanding.
- peteradio 5y ago
  I worked at a place with a lean 6 sigma certified specialist who towards the end of the companies doom effectively had the lead engineer cleaning out molding machines to track down every last tiny molded part that over the course of several years of continuous running had flung outside of its target. Same guy told me if the coke machine ever stole my change that he'd help me get it back from the vendor.
  - 7thaccount 5y ago
    All the lean sigma stuff seems like another useless management fad to me that only benefits consultants. Is that what you're saying here?
    - peteradio 5y ago
      I'm only speaking towards this one particularly useless buffoon, but the fact that he was allowed to wield any sort of power over anyone says something.
seanwilson 5y ago
> What McNamara didn’t keep track of was the narrative of the war, the meaning that it had both within the military forces of each side, but also in the civilian populations of the nations involved. I'm probably not following and not saying it's easy but aren't there some metrics you could track that with? How was it determined/measured later that the meaning was important? Couldn't you at least do polls in your own country?
- asplake 5y ago
  But that’s the point - it’s not just how many, but what they are willing and able to do, their positional and operational strengths and weaknesses, and so on. Condensing all that into a number is impossible.
NovemberWhiskey 5y ago
Or, put slightly more succinctly: "you get what you measure"
scubbo 5y ago
Surprised not to see Goodhart's Law[0] referenced here - "When a measure becomes a target, it ceases to be a good measure". Not the same concept, but a related one (as is the Cobra Effect[1], of which the poppy-field burning is an example) [0] https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law [1] https://en.wikipedia.org/wiki/Perverse_incentive#The_original_cobra_effect https://en.wikipedia.org/wiki/Perverse_incentive#The_origina...
- aidenn0 5y ago
  I also think that Goodhart's law is a stronger effect than the McNamara Fallacy for Afghanistan; when people are actively faking results, then that's not so much measuring the wrong things as just flat out failing at measuring things.
  - austincheney 5y ago
    The most important line from the article: Those who are concerned about falling into the trap of the McNamara Fallacy shouldn’t abandon quantitative measurements and metrics. A key example is Moneyball: https://en.m.wikipedia.org/wiki/Moneyball https://en.m.wikipedia.org/wiki/Moneyball In my line of work, JavaScript, people will intentionally ignore data that challenges their favored approach. Measuring things is important, but all these examples illustrate that measures are irrelevant when ethics are absent, which is worse than measures applied too narrowly as in the McNamara Fallacy.
- musicale 5y ago
  From [1]: "It was discovered that, by providing company executives with bonuses for reporting higher earnings, executives at Fannie Mae and other large corporations were encouraged to artificially inflate earnings statements and make decisions targeting short-term gains at the expense of long-term profitability." What a shocking revelation! Could it possibly apply to other companies that report quarterly results? ;-)
  - scubbo 5y ago
    I vividly remember a "small-hands" (smaller all-hands) very early in my career, where the VP basically said "we recognize that oncall is a burden, and we want to compensate you for it. At first we thought we should give you a bonus related to how many times you get paged - but then we realized that that's incentivizing you to build flaky systems. Then we considered giving a bonus in inverse relation to how often you're paged - but then we though, no, they're smart engineers, they'll just turn off the monitoring and collect the bonus. So we're giving you a flat rate". In the absence of some untamperable objective way to measure service health (the concept of SLAs was a distant dream at that point), can't fault that reasoning, tbh!
hwers 5y ago
At this point "fallacy" just strikes the same note to me as "misinformation". It's mostly used as a political device to get ideas you want to be true into social consensus but the reality is usually more ambiguous.
csours 5y ago
A different view of the problem: https://www.nngroup.com/articles/campbells-law/ https://www.nngroup.com/articles/campbells-law/
bell-cot 5y ago
"The moral is to the physical as three to one." - Napoleon I
beaconstudios 5y ago
> In another case of military metrics gone wrong, the US military reported success in undermining Taliban financing after it paid Afghan farmers to destroy their crops of opium poppies. What went unreported, however, was that the farmers planted larger fields of opium poppies in response, in the hopes that they might be paid by the US military to destroy the crops again. When US payments didn’t come through, the opium was harvested and entered the international drug trade. Much of the profit went to support the Taliban’s anti-American military operations. That's pretty ironic because its a near perfect replication of the Cobra effect, the canonical example for bad incentives leading to unintended outcomes: https://en.m.wikipedia.org/wiki/Perverse_incentive https://en.m.wikipedia.org/wiki/Perverse_incentive
serverlessmom 5y ago
"In war, whichever side may call itself the victor, there are no winners, but all are losers,"- Neville Chamberlain. This is a really interesting look at this type of mindset. I have often wondered what it is to "win" a war, and the quote that came to mind was the one I posted above. Measurement is certainly not understanding in all situations.
- hackeraccount 5y ago
  I think Clausewitz'saying "War is a mere continuation of politics by other means" was pretty close to the mark. Does anyone ever "win" politics? You have upsides and down and some things get pretty well settled but genuine victories are vanishingly rare whereas bitter tasting compromises seem to come about even when one side seems to hold all the cards.
sandworm101 5y ago
What is the name for the fallacy of thinking that wars are always meant to be won? Throughout history wars have been fought without any intention of "winning". Sometimes a war can serve a religious or ceremonial purpose, one that doesn't require a clear winner. Other wars have been fought for completely economic reasons. Others are proxy contests whereby greater powers can demonstrate their abilities without directly engaging each other. The false thinking is the assumption that participants always want or even care about winning.
- missedthecue 5y ago
  Wars might have unclear objectives and mission-creep, but I don't think anyone is fighting to lose.
  - serial_dev 5y ago
    The question is about what "winning" means for the people who start wars and keep fighting them. It might not be the same as for you. To quote Julian Assange, "The goal is to use Afghanistan to wash money out of the tax bases of the US and Europe through Afghanistan and back into the hands of a transnational security elite. The goal is an endless war, not a successful war".
    - missedthecue 5y ago
      I just don't buy that line of thinking at all. If that is truly their goal, there are way easier and subtle ways to do it without causing a collapse in political capital for the parties and politicians involved, which the Vietnam, Iraq, and Afghanistan wars did.
      - trasz 5y ago
        Such as? What other mechanism makes it possible to pay 700 billions per year to the private sector for something that the taxpayer can't see, because it's secret?
martincmartin 5y ago
Two examples come to mind: - Investing your money with the fund manager who has the highest returns over the last year, or even 5 years, is called "performance chasing" and generally has worse returns than investing in an index. Think investing in dot com stocks in 1999, and real estate in 2007. - In Moneyball, about using stats to improve a baseball team, they didn't care how many home runs someone hit. In fact, getting runs that way was considered bad. They wanted to get players on base and them get them home, even if through walks. There's a part of the book where a player is getting a lot of home runs, and therefore increasing the team's score, but the manager doesn't like it. "It's a process", "we have a process" he keeps saying.
johnp271 5y ago
Does the McNamara Fallacy have any application to our response to COVID? I often hear pundits of all sorts, medical doctors, epidemiologists, politicians, CDC scientists, etc, make statements such as "the data shows this" and "the data says that" and then follow up with "therefore the science says we must all do such-and-such". I hold a rather narrow, rigorous - maybe closed minded - opinion of what is 'science' (so to me 'social science' is an oxymoron) thus I have a degree of skepticism when data analysis is relied on to heavily for making conclusions that are then called 'scientific'.
- p_l 5y ago
  More like the data about COVID is used as fig leaf for metrics in other areas actually driving the decisions, or ideology/dogma. If you go with hard data and experience, you'd do hard moves like China (and many other asian countries did). In fact, similar moves have been done in the past in Europe (on the communist side of Iron Curtain) to stop epidemics, including even manual contact tracing. But because a non-trivial force in decision making has strong other incentives, and because of dogma like disbelief in aerosol transmission, we end up with really bad decisions with fig leaf of data analysis.
  - calyth2018 5y ago
    > because of dogma like disbelief in aerosol transmission It's not just that COVID can spread via aerosol. It's more so that the western world extrapolated the definition via a study on TB, neglecting that TB needed to infect deep in the lungs. https://www.wired.com/story/the-teeny-tiny-scientific-screwup-that-helped-covid-kill/ https://www.wired.com/story/the-teeny-tiny-scientific-screwu... There is a scientific paper version of this, btw.
- logicalmonster 5y ago
  > Does the McNamara Fallacy have any application to our response to COVID? In my opinion, yes. Many politicians and media are fixated on stats like "Covid cases" whereas society depends on multiple variables to function including the health of the economy, mental health and social cohesion of the population, people having hope for the future, the freedom to make choices for yourself, and people being able to exercise and not living unhealthy lives, etc. > "the data shows this" and "the data says that" This kind of objective data is generally science. > "therefore the science says we must all do such-and-such" This kind of thing is a policy or risk-management decision being wrongly conflated as science.
abhishekjha 5y ago
Does this have any QM implications?
deleted 5y ago
[deleted]
sf_rob 5y ago
This isn't a great article IMO (I say as someone who is a big fan of qualitative user research, mixed methods, etc). Per the first example, many types of attitudinal data can be quantified and conflating attitudinal data with qualitative data is itself a fallacy. It's possible that there were quantitative attitudinal signals that could have been captured or created as inputs to a more accurate model. Per the second example, this is more a question of data validity than the metric itself. If the metric could be validated through better design and gamification prevented then it would likely still be a helpful indicator. Granted this is a very hard problem.
- serial_dev 5y ago
  My thoughts, too. It's not that quantitative data is bad, he just picked the wrong one.
  - mcguire 5y ago
    That's the rub, ain't it? How do you know which is the right one beforehand?
    - serial_dev 5y ago
      That's a good point, you don't always know, so your quantitative data starts with qualitative (you decide what to measure). But, for the common cases you can follow the industry standard measurements. They also learned from the war question mentioned in the article, now they just measure public support, too.
shellback3 5y ago
It should be remembered that McNamara cut his teeth working for the office of statistics with Curtis LeMay in the use of B29 aircraft. He was proud that he was part of a larger effort using statistics and other math that helped to win the war and, with that hammer, most everything could be a nail.
shellback3 5y ago
McNamara cut his teeth on real world problems in the AAF's Office of Statistics and worked with General Curtis LaMay to plan how best to use the B29 bombers. He became an expert with statistics and other math tools and thought they could be widely applied to business and, of course, war. Once he had these marvelous tools he found nails everywhere. For instance the rate of German tank production was accurately estimated by collecting the serial numbers of all the tanks (and some tank components) that were knocked out. Best methods of attacking submarines was worked out this way as well as well as where to place armor in bombers, etc.
- topspin 5y ago
  > Once he had these marvelous tools he found nails everywhere. He also found two presidents and a sycophantic media that hung on his every word. His figures were excellent. In the earliest days of Vietnam, long before anyone outside of Asia could find it on a map, he produced eerily precise predictions of the costs -- in lives, dollars and time -- of the future conflict. He told them what a civil war in the jungle would look like and they pulled the trigger. As far as the value of measurement goes; I think most of the low hanging fruit has been picked (a consequence of the "Information Age") and what we struggle to 'measure' today is far less tractable. As a result our measurements are frequently corrupted in the service of prevailing agendas or not permitted at all for fear of undesirable results.
rossdavidh 5y ago
I believe the McNamara Fallacy is a real thing, although I would say Goodhart's Law expresses the more fundamental issue. But, the examples they give (Vietnam and Afghanistan wars) are bad examples, because in both cases the issue was primarily not that the U.S. military was relying too much on quantitative metrics, but rather that it was being asked to do a job which militaries are not good at. In neither case, despite what many deniers and apologists proclaim, was the war "winnable" by the military, because in both cases the primary problem was that the government of the nation in question did not have the respect (and thus support) of its people. There is no strategy that an outside military can use, which will fix this, because it's not a military problem, and a military has the wrong personnel, the wrong training, and thus the wrong institutional mindset. It is as if you sent the Red Cross in to defeat the Viet Cong or the Taliban, and then concluded that the reason they failed was that they were measuring numbers of hospital beds and counting how much medical equipment they needed. It was the political leadership, who decided this was a job for a military to perform, that was the root problem. McNamara's metrics were neither here nor there.
- paganel 5y ago
  > In neither case, despite what many deniers and apologists proclaim, was the war "winnable" by the military Basically, to win that war the US military should have been able to previously educate its politicians (i.e. the US politicians) and remind them that as "war is the continuation of politics by other means" so in countries like Afghanistan and Iraq a greater focus should have been given to politics before trying the war thing. After all that's what a guy like Clausewitz did, a general and military theorist whose work ended up educating lots and lots of politicians.
- jasonhansel 5y ago
  To quote C. Wright Mills, writing in 1960: > They know of no solutions to the paradoxes of the Middle East and Europe, the Far East and Africa except the landing of Marines. Being baffled, and also being very tired of being baffled, they have come to believe that there is no way out—except war—which would remove all the bewildering paradoxes of their tedious and now misguided attempts to construct peace. In place of these paradoxes they prefer the bright, clear problems of war—as they used to be. For they still believe that "winning" means something, although they never tell us what.
skybrian 5y ago
Maybe a better way of thinking about it is that a process that gathers numbers can be useful, but understanding the context in which the numbers were gathered (methodology and whatever else is happening at the time) is more important than the numbers themselves. Without context, you don't know what you have. For example, surveys gather numbers, but if you don't understand how the people who answered the questions interpreted them, you don't know what the numbers mean. Asking people what they really meant by their answer is only possible for the people doing the survey.
jiggawatts 5y ago
Can someone tell Microsoft this?
jupp0r 5y ago
TDLR: quantitative analysis is necessary, but not sufficient for success
nitwit005 5y ago
These military positions require "not understanding" as a prerequisite. Do you think you'd be put in charge of something like the war in Vietnam, if you didn't tell the president you could win the war for them? You'd just have been sidelined or fired if you gave a realistic opinion. Which is not unique to the military, of course. Have fun letting the CEO know their sales goals are unobtainable.
zeruch 5y ago
I just wrote about this (really briefly) on LinkedIn a week or so ago, because I've seen a growing trend towards "management by spreadsheet" and I think its an enormous error in terms of team leadership/management. Much as jscode says in the comments: "The moral of the story is that people with analyst mindsets play an essential role in our economy, but sometimes giving those people power over large organizations can have disastrous consequences. There truly is a disconnect between measurement and understanding. "
jwsteigerwalt 5y ago
The core premise of the McNamara fallacy is not wrong. However, this article makes a critical mistake about the origin. McNamara was not just a business executive that believed he could take his knowledge to the Defense Department. He was an officer in the Army Air Force Statistical Control office that organized the Air Force during World War II into the successful machine that it became. The fallacy is not that success in business is transferable to war, The fallacy is that the successes in World War II were transferable to Vietnam.
gilgoomesh 5y ago
The article attacks measurement based approaches, in general, which I don't think is supported by the anecdote of McNamara's failings in Vietnam. Kill count was simply a terrible metric of victory. Vietnam was a civil war, which requires a political victory. This means allegiance of local leaders and populist support – both of which are measurable values. A kill count is often inversely correlated with political support. McNamara's failing here is more like the XY problem [1]. He incorrectly assumed that a brutal display of military power was the way to political victory and so he measured the wrong value. [1] - https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem
hristov 5y ago
A. Please do not center justify paragraphs, this is really annoying and makes things very hard to read. B. It is very tricky to make conclusions about life and business from US conquering military campaigns from within the US, because these campaigns have been shrouded with so many layers of lies self delusion and bullshit, that a lot of the generally accepted facts are just not true, and thus not able to teach us anything. By the way, I do not think America is in any way unique about this. Every country's conquering wars are carefully explained with heaps of bullshit from that country. There are some very rare exception as in with Germany and Japan and WW2, but usually every country carefully shapes the official story to make it so that the conquering wars are not completely based on greed and lust for power, when they usually are. So lets talk about McNamara and Vietnam. The problems there had little to do with some kind of duality between the quantitative and qualitative. In fact if McNamara honestly followed quantitative analysis, he would reach very different conclusions about Vietnam. If, for example, he commissioned a very simple very quantitative opinion poll among Vietnamese about whether they would prefer to be ruled by a puppet regime installed by a foreign country or by the independent patriotic leader that just defeated the French colonialists, he would see that the official reason for invading Vietnam was just not valid. It is not clear whether such a poll has been commissioned but McNamara probably knew what the results would be, because the US never permitted a free election in South Vietnam. Thus, faced with a simple quantitative fact against the war, the quantitative managerial McNamara ignored it and suppressed it. So no it is not about the quantitative vs the qualitative, it is the usual problem with wars of conquest by rich countries. People in positions that would benefit from the war carefully and persistently create a web of lies and delusions that is intended to drive the country to war.
- dundarious 5y ago
  I think the most HN-relevant elements of the fallacy are not about quantitative vs. qualitative. > The McNamara Fallacy is to presume that (A) quantitative models of reality are always more accurate than other models; (B) the quantitative measurements that can be made most easily must be the most relevant; and (C) factors other than those currently being used in quantitative metrics must either not exist or not have a significant influence on success. (A) is quant vs. qual, but (B) and (C) are about fooling yourself into thinking you're doing something useful just because you're quantitative. And while I agree with pretty much all your commentary about the war, I think this would be a very qualitative (not quantitative) assessment on McNamara's part: > It is not clear whether such a poll has been commissioned but McNamara probably knew what the results would be, because the US never permitted a free election in South Vietnam. Thus, faced with a simple quantitative fact against the war, the quantitative managerial McNamara ignored it and suppressed it.
morelandjs 5y ago
I don’t really understand the narrow definition of quantitative analysis referenced here and elsewhere. If you know your metrics are failing you, then you implicitly have a more advanced model of success. Define this more advanced mental model so it can be invalidated. If your mental model is not defined narrowly enough to be invalidated, then how can you improve upon it? I think the lesson here was that the measure of success was defined poorly, not that quantitative analysis is flawed a practice.
lvass 5y ago
I'm amazed they didn't mention McNamara's 100,000. There's so much to learn trying to not repeat this guy's mistakes. Project 100,000 is an enlightening piece of history, to say the least.
sloan 5y ago
“There’s the old apocryphal story that in 1967, they went to the basement of the Pentagon, when the mainframe computers took up the whole basement, and they put on the old punch cards everything you could quantify. Numbers of ships, numbers of tanks, numbers of helicopters, artillery, machine gun, ammo—everything you could quantify,” says James Willbanks, the chair of military history at U.S. Army Command and General Staff College. “They put it in the hopper and said, ‘When will we win in Vietnam?’ They went away on Friday and the thing ground away all weekend. [They] came back on Monday and there was one card in the output tray. And it said, 'You won in 1965.’” https://www.theatlantic.com/technology/archive/2017/10/the-computer-that-predicted-the-us-would-win-the-vietnam-war/542046/ https://www.theatlantic.com/technology/archive/2017/10/the-c...
craigmcnamara 5y ago
Tough to share a last name with this guy.
- mhh__ 5y ago
  Even stranger if it was the middle name too
_8j50 5y ago
The US did not lose the war against the VC, they stopped supporting south vietnam's war against the VC and withdrew because the war became unpopular. The distinction is important because ultimately it wasn't America's war and the VC did not overwhelm the US military into surrender. It was the civilian government of the US that directed an end of their involvement. If anything, China won a cold war against the US who feared a direct conflict and caved in. With Afghanistan, the war against the taliban government was won 15+ years ago. The US thought it could play nation-builder like with Japan. The continued involvement in afghanistan was part of the ongoing "war against terror" to reduce the chance of afghanistan becoming a heaven for terrorists again. The war against afghanistan was won, what the US lost was a prolonged occupation. If you look at history, the victor would more or less burn the invaded country to the ground and enslave its people to avoid similar long term issues, one can argue it was both self-interest and an ideology of being humane to civilians that resulted in the waste of post-war afghanistan occupation. Regarding metrics, I am fighting this right now. I found out what is expected of me early on and worked to extract that value out of myself and support my team but in the end my struggle is being able to explain metrics that reflect a years worth of work that only capture at best 30% of what is expected of me. Either way, can't complain, management isn't easy but meeting metrics is. what I dislike is if I have good metrics and stop there, it isn't enough. Metrics will always set a maximum ceiling on quality, if management can accept that I have no problem with metrics driven performance appraisal so long as it pays.
- aiilns 5y ago
  > The US thought it could play nation-builder like with Japan. The continued involvement in afghanistan was part of the ongoing "war against terror" to reduce the chance of afghanistan becoming a heaven for terrorists again. I would more easily characterize US external politics as nation-destroyer than nation-builder. Even if you disregard the evidence, it's quite a stretch to believe that the situation in Afghanistan is solely due to incompetence despite US good intentions. > If you look at history, the victor would more or less burn the invaded country to the ground and enslave its people to avoid similar long term issues Even if this has happened with some frequency I'm not so sure it's the rule; certainly there are many exceptions. In any case, it's no argument for doing the same.
toss1 5y ago
Drunk person, near streetlight, lost key ... Of course it is much easier to manage things that you can measure. That does NOT mean that if you cannot quantitatively measure it, that it can't be managed It certainly does not mean that if you haven't yet developed an effective metric, that it does not matter -- it's just mind-blowing that McNamara went that far. The way to do it is to FIRST figure out what matters. Then figure out if it can be measured, and if not, how to qualitatively assess it. Then also figure out how to ensure that the metrics don't become the goal (with perhaps the sole exception being if you are in sports and the goal is winning races/games, so that's a pretty good metric in a suite of metrics).

365 points by wenc 5y ago

deleted 5y ago

[deleted]

From a game-theory POV, all wars are caused by misunderstandings / partial information. If each side really knew the true strength of the other side, it would be clear which would win and therefore actual war is illogical and unnecessary.

hackeraccount 5y ago
Is that true? What if the situation is that I am currently stronger then you but can expect to become weaker in the future? Why not force a war now if I can expect to win? I still haven't finished it (too depressing) but my reading of the book The Sleepwalkers about the events leading up to WWI is that all the parties thought they were in that place; even if they all had a full picture it's certainly possible those beliefs would in fact be true.
- aeternum 5y ago
  If I know that you are currently stronger and would ultimately win a war, I should logically concede now since I will lose anyway. If I know that you are currently stronger but we both also know that I can draw out a war such that I eventually become stronger, then you shouldn't go to war since you will ultimately lose.
  - vkou 5y ago
    > If I know that you are currently stronger and would ultimately win a war, I should logically concede now since I will lose anyway. You'll often lose less if you fight to a loss. In fact, you often won't have to fight at all if you make it clear to your adversary that you will fight to a loss. If I'm a street thug, I'm not going to try to mug someone who clearly communicates that their response to a mugging is detonating their suicide vest. I will, however, repeatedly victimize someone who clearly communicate that they are a pushover. This is an iterative game, against the same players. Loss aversion is not a winning strategy to pursue in the long-term. Something resembling tit-for-tat is.
vlovich123 5y ago
That's a fallacious line of reasoning on several fronts. A) The beneficiaries of a war need not be the ones who pay the cost. If you personally stand to benefit from the war itself, regardless of win or lose, you want the war. B) It's applying a rational economics approach when it's pretty clear that humans aren't rational. Japan kept fighting even when it was clear they would lose. Pride, image, your future after the war. All things that impact the decision that gets made. C) War isn't typically total like it was in WWII. Most wars are just skirmishes or prolonged skirmishes that don't meaningfully change borders. In such a scenario, a single "war" loss may still be an important fight to have in the middle of the overall conflict (i.e. to signal to your opponent that you're not a pushover and that continuing on their current path has consequences). D) Win/lose isn't the only outcome of war. Lose/lose is also another scenario. Additionally, you might lose the war but winning may not be the goal. For example, maybe it only takes me 1/10th the resources for me to wage the war that it takes for you to win. A single war might cost me 100 units, but it's important for me to be able to cause you to spend 1000 units because then I have a 900 unit advantage in some other area. There's many more, but I don't think imperfect information is the only reason war exists.
- tomrod 5y ago
  (1) Externalities are handled in the game theory literature (2) Preference definitions are not in scope for most game theory lit; what you're describing are preferences and expected utility, which ARE in the literature and metamodel (3) This is not really true. It depends on how big you are and how big your coalition is. Consider that Saddam Hussain is no longer in charge of Iraq. (4) Yes, potential outcomes are considered under game theory. I get your point that OP's statement feels reductionist, but I wanted to explain that the OP is actually correct.
  - mcguire 5y ago
    You didn't work at Rand in the 1960s, did you? :-) Anyway, the OP may be correct, but is also completely useless since it is impossible to know your own true strength, much less that of the other side.
13415 5y ago
Wait a minute. Shouldn't the goal of the defender be to make it clear to the attacker that they would incur losses that outweigh possible wins for the attacker, thereby making the attack a lose-lose scenario? This is only possible if the defender makes it credible to the potential attacker that they will defend their country even when the costs would be extremely high and they cannot possibly win the conflict. In other words, at least from deterrence point of view it's not about winning, it's about making sure the attacker would overall lose more than gain (which is not the same as winning against the attacker).
crazy1van 5y ago
> If each side really knew the true strength of the other side, it would be clear which would win This is too simplistic. Often wars are heavily influenced by things other than just the strengths of both sides. For example - What if the ground hadn't been soaked from days of rain at Agincourt and Waterloo? What if the Germans hadn't held back their armor reinforcements for so long on D-Day?
- asdff 5y ago
  They'd still lose. Wars are fought not from individual battles but from logistics. Modern military's don't even engage if there isn't a lopsided power imbalance favoring their success.
  - 0xdeadbeefbabe 5y ago
    Yeah like in Black Hawk Down.
    - asdff 5y ago
      In real life the battle of mogadishu saw Americans with 10x fewer casualties. Most modern battles have numbers like this with an order of magnitude fewer casualties on the American side, because American logistic planners try to ensure a massive power advantage.
      - mcguire 5y ago
        And yet...
        asdff 5y ago
        And yet what? Are you not convinced by the casualty data?
        mcguire 5y ago
        I don't seem to recall Mogadishu being any kind of "victory". It's almost like the casualty data didn't tell the whole story.
s28l 5y ago
There is a lot to disagree with here. First, the article itself provides an example of an army with a superior fighting force that had far fewer casualties than the other side, yet still lost the war. That was due to the civilians back at home lacking the stomach for a long, protracted war abroad. So even if one side has superior strength, and even if the side with the superior strength wins every battle, they might still lose the war. Another issue is your implicit assumption that the side with the superior fighting force will always win the war, but I don't think you can make that assumption. Just like the better football team can lose to the underdog, there is a stochastic element to warfare. A sudden bit of bad weather can turn the tide of a battle. There are countless other elements that are unobservable and unpredictable that can decide which side wins. There are also asymmetric payoffs to going to war. A nation might have a slim chance of victory, but the cost of defeat or surrender might be genocide or subjection. How do you assign a payoff to the choice "surrender" when the outcome is the destruction of your nation as it once existed? [0] https://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_and_mixed_strategies https://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_an...
aeturnum 5y ago
This seems untrue. There are also genuine uncertainties about the future: things that neither side knows but will impact the outcome. Because of that, there is always a range of outcomes for a conflict, and so both sides would still have an incentive to carry out the war. Consider, for instance, if there are two countries (A and B) looking at conflict with perfect information. Given their relative strengths, the war will cost both $1 billion, but A will win and reap $1 billion + $1 in plunder. In your model, B will always surrender and give A $1, but the reality is there is a big spread about both the costs and the gains. Lots depends on how well each side fights and responds, even with perfect knowledge of the other. So...you know, if it's the US versus...the Philippines[1], then sure, the uncertain range of outcomes is small enough that rational thing for the Philippines to do is surrender. But for countries that are even reasonably well-matched, they would want to fight with the goal of making the conflict unattractive for the other side. Probably the best historical example of this is the USSR in Afghanistan. [1] I am sure the US would never invade and occupy them just because the US found it convenient ;)
- thaumasiotes 5y ago
  > Consider, for instance, if there are two countries (A and B) looking at conflict with perfect information. Given their relative strengths, the war will cost both $1 billion, but A will win and reap $1 billion + $1 in plunder. In your model, B will always surrender and give A $1 In the perfect information scenario, B will surrender more than $1. B is better off at any tribute level up to $1,000,000,000.
  - aeturnum 5y ago
    Sure, I didn't go far into detail, but it's actually worse than either of us has said because once B surrenders anything ($1 or $1,000,000,000), they are at even greater disadvantage and "logically" should surrender more money. In fact, they should surrender the entire country, as once they are at a disadvantage A can keep coming back forever. Effectively, the idea that war would not happen with perfect information suggests that the logical thing to do is for all countries to surrender control to the strongest country immediately.
chasd00 5y ago
i'll add that there's a lot of people that see death as the door to paradise and fighting for their cause as the key to that door. No amount of logic or information is going to stop them. when you're dealing with the mentally ill, all bets are off.
deleted 5y ago
[deleted]

paulpauper 5y ago

I don't understands what the fallacy is. It is that an overreliance on data can overlook factors not in the data? Why is that a surprise. You would also have to show that not relying on data would generate better results.

warning26 5y ago
> It is that an overreliance on data can overlook factors not in the data? Why is that a surprise? The vast majority of PMs I've worked with don't seem to understand this at all. If it's not in the metrics, it doesn't exist!
0xdeadbeefbabe 5y ago
If understanding is what you seek then https://www.youtube.com/watch?v=eNDsd798HR4 https://www.youtube.com/watch?v=eNDsd798HR4 is a nice long discussion on McNamara overlooking factors not in the data. The fallacy didn't earn its name for simple reasons.
_Nat_ 5y ago
It's probably one of those things that might seem more obvious in simple cases, but might surprise folks in more complex cases. For example, a simple case: Say you go on vacation for a few weeks with a certain amount of cash to spend. Upon arriving at your destination, you immediately purchase some indulgence, and, hey, you're feeling better! Why not immediately keep spending as much as possible to maximize? For example, a complex case: Say you're leading a country. You set up policies that, after a few years, seem to have led to a higher GDP than was previously expected. Does that suggest that the policies are leading to a better future? In the simple cases, like the vacation-example above, it's easy enough to understand the scenario and what's going on. And we can imagine that, hey, immediately spending all of the cash might lead to poor consequences for the rest of the vacation, even if they display some quick-satisfaction at first. But in more complex cases, like with a country's policies leading to a higher GDP, stuff can get trickier. We might say that it's the lack of a top-level model: unlike in the vacation scenario, where we were easily able to predict that there'd be a lack of money later in the vacation, it might be harder to say what else might be going on besides the GDP going up. And all other things held equal, presumably a higher GDP would be better than a lower GDP, and therefore all evidence points toward the policies being a good idea, right?

dempedempe 5y ago

This article is good, but not great - the author only gives one example of how quantitative-only reasoning can be bad (the example of the poppies). The other "example" is just the US military lying. There are also no specific examples of non-quantitative reasoning that, if ignored, would be damaging. I feel like the Wikipedia article does a better job explaining this: https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy Also, why is there an entire webpage dedicated to this?

enkid 5y ago
The example with the US military lying certainly demonstrates another issue with over valuing metrics - making sure you have good data. People who get wrapped around metrics also tend to not look at the data they are being presented. One of the issues McNamara had was he was receiving inflated body count figures. [0] Not only was he measuring the wrong thing, he was measuring it badly. [0] https://en.wikipedia.org/wiki/Vietnam_War_body_count_controversy https://en.wikipedia.org/wiki/Vietnam_War_body_count_controv...
- the_af 5y ago
  It's also a good example of perverse incentives or "gaming the metrics" (in a horrifying way). If the main measure of success was body count, and every kill was considered an enemy by default, this encouraged just killing people and counting them as successes.
  - shellback3 5y ago
    I was in Vietnam in a Naval Support Group. I had conversations with army officers that had combat experience and learned that since no statistics were collected about dead civilians all the dead were counted as enemy soldiers - and everyone from the top down knew it.
hummusandsushi 5y ago
The webpage appears to be made to serve as a warning to data-driven businesses to not fall into the same perverse set of incentives that McNamara created, to encourage business managers to diversify their accounts of the success of their business beyond just the quantitative narrative.

snidane 5y ago

I wonder if it gets widespread to oppose the dominant hyperquantitative, every fart tracked in jira, can't measure - can't manage trend. Which was never pushed by anybody from the industry, but got spread by project management frauds. https://deming.org/myth-if-you-cant-measure-it-you-cant-manage-it/ https://deming.org/myth-if-you-cant-measure-it-you-cant-mana...

eumoria 5y ago

McNamara basically admits this himself in the documentary The Fog of War: https://en.wikipedia.org/wiki/The_Fog_of_War https://en.wikipedia.org/wiki/The_Fog_of_War It's worth a watch but it's very soft on him and his role. Still a very good documentary.

ProAm 5y ago
Great documentary. I felt it was a person trying to come clean and ease his conscious on his death bed.
- 2OEH8eoCRo0 5y ago
  Agreed. Great documentary. The part where Castro said he urged Russia to launch their nukes from Cuba to the US knowing it would destroy Cuba was chilling. Humans are not always logical. Don't assume somebody wont drag an entire country or the world to total destruction for some deranged cause. I didn't get the death bed vibe from McNamara but I definitely felt that he was genuinely reflecting on the past. The documentary on Rumsfeld was the polar opposite. I could also see Rumsfeld not wanting to give the enemy of an ongoing conflict any shred of material. It makes for a less interesting documentary.
  - nickdothutton 5y ago
    Morris himself said something like he didnt really feel he got to know Rumsfeld and had no real idea what was in his mind compared to his previous interview with McNamara.
    - wayoutthere 5y ago
      Guess Donald himself was one of the “unknown unknowns”
      - codechad 5y ago

jscode 5y ago

Quick story: I was the CFO for a company that sold to a private equity group (PEG). I took over as the CEO as the founders retired, leaving me to deal with the PEG. It quickly became apparent that the PEG managers looked at everything through the lens of an Excel spreadsheet. These guys were brilliant attorneys and analysts but lacked experience building businesses and managing teams. Ultimately, they couldn’t add much value in terms of operations or strategy, but they were great at financial modeling/quantitative analysis and forcing us to justify expenses. That may sound good at first—eliminating wasteful spending—but it ultimately led to the gradual erosion of the company culture and employee loyalty. It’s easy to cut benefits and pay given that many workers lack the leverage to do anything about it, while it’s much harder to reduce hard costs like materials and equipment. That meant employees just kept getting squeezed, and it was surprisingly difficult to quantify the impact that terminating an employee or cutting benefits would have on morale/culture/performance. The moral of the story is that people with analyst mindsets play an essential role in our economy, but sometimes giving those people power over large organizations can have disastrous consequences. There truly is a disconnect between measurement and understanding.

peteradio 5y ago
I worked at a place with a lean 6 sigma certified specialist who towards the end of the companies doom effectively had the lead engineer cleaning out molding machines to track down every last tiny molded part that over the course of several years of continuous running had flung outside of its target. Same guy told me if the coke machine ever stole my change that he'd help me get it back from the vendor.
- 7thaccount 5y ago
  All the lean sigma stuff seems like another useless management fad to me that only benefits consultants. Is that what you're saying here?
  - peteradio 5y ago
    I'm only speaking towards this one particularly useless buffoon, but the fact that he was allowed to wield any sort of power over anyone says something.

seanwilson 5y ago

> What McNamara didn’t keep track of was the narrative of the war, the meaning that it had both within the military forces of each side, but also in the civilian populations of the nations involved. I'm probably not following and not saying it's easy but aren't there some metrics you could track that with? How was it determined/measured later that the meaning was important? Couldn't you at least do polls in your own country?

asplake 5y ago
But that’s the point - it’s not just how many, but what they are willing and able to do, their positional and operational strengths and weaknesses, and so on. Condensing all that into a number is impossible.

NovemberWhiskey 5y ago

Or, put slightly more succinctly: "you get what you measure"

scubbo 5y ago

Surprised not to see Goodhart's Law[0] referenced here - "When a measure becomes a target, it ceases to be a good measure". Not the same concept, but a related one (as is the Cobra Effect[1], of which the poppy-field burning is an example) [0] https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law [1] https://en.wikipedia.org/wiki/Perverse_incentive#The_original_cobra_effect https://en.wikipedia.org/wiki/Perverse_incentive#The_origina...

aidenn0 5y ago
I also think that Goodhart's law is a stronger effect than the McNamara Fallacy for Afghanistan; when people are actively faking results, then that's not so much measuring the wrong things as just flat out failing at measuring things.
- austincheney 5y ago
  The most important line from the article: Those who are concerned about falling into the trap of the McNamara Fallacy shouldn’t abandon quantitative measurements and metrics. A key example is Moneyball: https://en.m.wikipedia.org/wiki/Moneyball https://en.m.wikipedia.org/wiki/Moneyball In my line of work, JavaScript, people will intentionally ignore data that challenges their favored approach. Measuring things is important, but all these examples illustrate that measures are irrelevant when ethics are absent, which is worse than measures applied too narrowly as in the McNamara Fallacy.
musicale 5y ago
From [1]: "It was discovered that, by providing company executives with bonuses for reporting higher earnings, executives at Fannie Mae and other large corporations were encouraged to artificially inflate earnings statements and make decisions targeting short-term gains at the expense of long-term profitability." What a shocking revelation! Could it possibly apply to other companies that report quarterly results? ;-)
- scubbo 5y ago
  I vividly remember a "small-hands" (smaller all-hands) very early in my career, where the VP basically said "we recognize that oncall is a burden, and we want to compensate you for it. At first we thought we should give you a bonus related to how many times you get paged - but then we realized that that's incentivizing you to build flaky systems. Then we considered giving a bonus in inverse relation to how often you're paged - but then we though, no, they're smart engineers, they'll just turn off the monitoring and collect the bonus. So we're giving you a flat rate". In the absence of some untamperable objective way to measure service health (the concept of SLAs was a distant dream at that point), can't fault that reasoning, tbh!

hwers 5y ago

At this point "fallacy" just strikes the same note to me as "misinformation". It's mostly used as a political device to get ideas you want to be true into social consensus but the reality is usually more ambiguous.

csours 5y ago

A different view of the problem: https://www.nngroup.com/articles/campbells-law/ https://www.nngroup.com/articles/campbells-law/

bell-cot 5y ago

"The moral is to the physical as three to one." - Napoleon I

beaconstudios 5y ago

> In another case of military metrics gone wrong, the US military reported success in undermining Taliban financing after it paid Afghan farmers to destroy their crops of opium poppies. What went unreported, however, was that the farmers planted larger fields of opium poppies in response, in the hopes that they might be paid by the US military to destroy the crops again. When US payments didn’t come through, the opium was harvested and entered the international drug trade. Much of the profit went to support the Taliban’s anti-American military operations. That's pretty ironic because its a near perfect replication of the Cobra effect, the canonical example for bad incentives leading to unintended outcomes: https://en.m.wikipedia.org/wiki/Perverse_incentive https://en.m.wikipedia.org/wiki/Perverse_incentive

serverlessmom 5y ago

"In war, whichever side may call itself the victor, there are no winners, but all are losers,"- Neville Chamberlain. This is a really interesting look at this type of mindset. I have often wondered what it is to "win" a war, and the quote that came to mind was the one I posted above. Measurement is certainly not understanding in all situations.

hackeraccount 5y ago
I think Clausewitz'saying "War is a mere continuation of politics by other means" was pretty close to the mark. Does anyone ever "win" politics? You have upsides and down and some things get pretty well settled but genuine victories are vanishingly rare whereas bitter tasting compromises seem to come about even when one side seems to hold all the cards.

sandworm101 5y ago

What is the name for the fallacy of thinking that wars are always meant to be won? Throughout history wars have been fought without any intention of "winning". Sometimes a war can serve a religious or ceremonial purpose, one that doesn't require a clear winner. Other wars have been fought for completely economic reasons. Others are proxy contests whereby greater powers can demonstrate their abilities without directly engaging each other. The false thinking is the assumption that participants always want or even care about winning.

missedthecue 5y ago
Wars might have unclear objectives and mission-creep, but I don't think anyone is fighting to lose.
- serial_dev 5y ago
  The question is about what "winning" means for the people who start wars and keep fighting them. It might not be the same as for you. To quote Julian Assange, "The goal is to use Afghanistan to wash money out of the tax bases of the US and Europe through Afghanistan and back into the hands of a transnational security elite. The goal is an endless war, not a successful war".
  - missedthecue 5y ago
    I just don't buy that line of thinking at all. If that is truly their goal, there are way easier and subtle ways to do it without causing a collapse in political capital for the parties and politicians involved, which the Vietnam, Iraq, and Afghanistan wars did.
    - trasz 5y ago
      Such as? What other mechanism makes it possible to pay 700 billions per year to the private sector for something that the taxpayer can't see, because it's secret?

martincmartin 5y ago

Two examples come to mind: - Investing your money with the fund manager who has the highest returns over the last year, or even 5 years, is called "performance chasing" and generally has worse returns than investing in an index. Think investing in dot com stocks in 1999, and real estate in 2007. - In Moneyball, about using stats to improve a baseball team, they didn't care how many home runs someone hit. In fact, getting runs that way was considered bad. They wanted to get players on base and them get them home, even if through walks. There's a part of the book where a player is getting a lot of home runs, and therefore increasing the team's score, but the manager doesn't like it. "It's a process", "we have a process" he keeps saying.

johnp271 5y ago

Does the McNamara Fallacy have any application to our response to COVID? I often hear pundits of all sorts, medical doctors, epidemiologists, politicians, CDC scientists, etc, make statements such as "the data shows this" and "the data says that" and then follow up with "therefore the science says we must all do such-and-such". I hold a rather narrow, rigorous - maybe closed minded - opinion of what is 'science' (so to me 'social science' is an oxymoron) thus I have a degree of skepticism when data analysis is relied on to heavily for making conclusions that are then called 'scientific'.

p_l 5y ago
More like the data about COVID is used as fig leaf for metrics in other areas actually driving the decisions, or ideology/dogma. If you go with hard data and experience, you'd do hard moves like China (and many other asian countries did). In fact, similar moves have been done in the past in Europe (on the communist side of Iron Curtain) to stop epidemics, including even manual contact tracing. But because a non-trivial force in decision making has strong other incentives, and because of dogma like disbelief in aerosol transmission, we end up with really bad decisions with fig leaf of data analysis.
- calyth2018 5y ago
  > because of dogma like disbelief in aerosol transmission It's not just that COVID can spread via aerosol. It's more so that the western world extrapolated the definition via a study on TB, neglecting that TB needed to infect deep in the lungs. https://www.wired.com/story/the-teeny-tiny-scientific-screwup-that-helped-covid-kill/ https://www.wired.com/story/the-teeny-tiny-scientific-screwu... There is a scientific paper version of this, btw.
logicalmonster 5y ago
> Does the McNamara Fallacy have any application to our response to COVID? In my opinion, yes. Many politicians and media are fixated on stats like "Covid cases" whereas society depends on multiple variables to function including the health of the economy, mental health and social cohesion of the population, people having hope for the future, the freedom to make choices for yourself, and people being able to exercise and not living unhealthy lives, etc. > "the data shows this" and "the data says that" This kind of objective data is generally science. > "therefore the science says we must all do such-and-such" This kind of thing is a policy or risk-management decision being wrongly conflated as science.

abhishekjha 5y ago

Does this have any QM implications?

deleted 5y ago

[deleted]

sf_rob 5y ago

This isn't a great article IMO (I say as someone who is a big fan of qualitative user research, mixed methods, etc). Per the first example, many types of attitudinal data can be quantified and conflating attitudinal data with qualitative data is itself a fallacy. It's possible that there were quantitative attitudinal signals that could have been captured or created as inputs to a more accurate model. Per the second example, this is more a question of data validity than the metric itself. If the metric could be validated through better design and gamification prevented then it would likely still be a helpful indicator. Granted this is a very hard problem.

serial_dev 5y ago
My thoughts, too. It's not that quantitative data is bad, he just picked the wrong one.
- mcguire 5y ago
  That's the rub, ain't it? How do you know which is the right one beforehand?
  - serial_dev 5y ago
    That's a good point, you don't always know, so your quantitative data starts with qualitative (you decide what to measure). But, for the common cases you can follow the industry standard measurements. They also learned from the war question mentioned in the article, now they just measure public support, too.

shellback3 5y ago

It should be remembered that McNamara cut his teeth working for the office of statistics with Curtis LeMay in the use of B29 aircraft. He was proud that he was part of a larger effort using statistics and other math that helped to win the war and, with that hammer, most everything could be a nail.

shellback3 5y ago

McNamara cut his teeth on real world problems in the AAF's Office of Statistics and worked with General Curtis LaMay to plan how best to use the B29 bombers. He became an expert with statistics and other math tools and thought they could be widely applied to business and, of course, war. Once he had these marvelous tools he found nails everywhere. For instance the rate of German tank production was accurately estimated by collecting the serial numbers of all the tanks (and some tank components) that were knocked out. Best methods of attacking submarines was worked out this way as well as well as where to place armor in bombers, etc.

topspin 5y ago
> Once he had these marvelous tools he found nails everywhere. He also found two presidents and a sycophantic media that hung on his every word. His figures were excellent. In the earliest days of Vietnam, long before anyone outside of Asia could find it on a map, he produced eerily precise predictions of the costs -- in lives, dollars and time -- of the future conflict. He told them what a civil war in the jungle would look like and they pulled the trigger. As far as the value of measurement goes; I think most of the low hanging fruit has been picked (a consequence of the "Information Age") and what we struggle to 'measure' today is far less tractable. As a result our measurements are frequently corrupted in the service of prevailing agendas or not permitted at all for fear of undesirable results.

rossdavidh 5y ago

I believe the McNamara Fallacy is a real thing, although I would say Goodhart's Law expresses the more fundamental issue. But, the examples they give (Vietnam and Afghanistan wars) are bad examples, because in both cases the issue was primarily not that the U.S. military was relying too much on quantitative metrics, but rather that it was being asked to do a job which militaries are not good at. In neither case, despite what many deniers and apologists proclaim, was the war "winnable" by the military, because in both cases the primary problem was that the government of the nation in question did not have the respect (and thus support) of its people. There is no strategy that an outside military can use, which will fix this, because it's not a military problem, and a military has the wrong personnel, the wrong training, and thus the wrong institutional mindset. It is as if you sent the Red Cross in to defeat the Viet Cong or the Taliban, and then concluded that the reason they failed was that they were measuring numbers of hospital beds and counting how much medical equipment they needed. It was the political leadership, who decided this was a job for a military to perform, that was the root problem. McNamara's metrics were neither here nor there.

paganel 5y ago
> In neither case, despite what many deniers and apologists proclaim, was the war "winnable" by the military Basically, to win that war the US military should have been able to previously educate its politicians (i.e. the US politicians) and remind them that as "war is the continuation of politics by other means" so in countries like Afghanistan and Iraq a greater focus should have been given to politics before trying the war thing. After all that's what a guy like Clausewitz did, a general and military theorist whose work ended up educating lots and lots of politicians.
jasonhansel 5y ago
To quote C. Wright Mills, writing in 1960: > They know of no solutions to the paradoxes of the Middle East and Europe, the Far East and Africa except the landing of Marines. Being baffled, and also being very tired of being baffled, they have come to believe that there is no way out—except war—which would remove all the bewildering paradoxes of their tedious and now misguided attempts to construct peace. In place of these paradoxes they prefer the bright, clear problems of war—as they used to be. For they still believe that "winning" means something, although they never tell us what.

skybrian 5y ago

Maybe a better way of thinking about it is that a process that gathers numbers can be useful, but understanding the context in which the numbers were gathered (methodology and whatever else is happening at the time) is more important than the numbers themselves. Without context, you don't know what you have. For example, surveys gather numbers, but if you don't understand how the people who answered the questions interpreted them, you don't know what the numbers mean. Asking people what they really meant by their answer is only possible for the people doing the survey.

jiggawatts 5y ago

Can someone tell Microsoft this?

jupp0r 5y ago

TDLR: quantitative analysis is necessary, but not sufficient for success

nitwit005 5y ago

These military positions require "not understanding" as a prerequisite. Do you think you'd be put in charge of something like the war in Vietnam, if you didn't tell the president you could win the war for them? You'd just have been sidelined or fired if you gave a realistic opinion. Which is not unique to the military, of course. Have fun letting the CEO know their sales goals are unobtainable.

zeruch 5y ago

I just wrote about this (really briefly) on LinkedIn a week or so ago, because I've seen a growing trend towards "management by spreadsheet" and I think its an enormous error in terms of team leadership/management. Much as jscode says in the comments: "The moral of the story is that people with analyst mindsets play an essential role in our economy, but sometimes giving those people power over large organizations can have disastrous consequences. There truly is a disconnect between measurement and understanding. "

jwsteigerwalt 5y ago

The core premise of the McNamara fallacy is not wrong. However, this article makes a critical mistake about the origin. McNamara was not just a business executive that believed he could take his knowledge to the Defense Department. He was an officer in the Army Air Force Statistical Control office that organized the Air Force during World War II into the successful machine that it became. The fallacy is not that success in business is transferable to war, The fallacy is that the successes in World War II were transferable to Vietnam.

gilgoomesh 5y ago

The article attacks measurement based approaches, in general, which I don't think is supported by the anecdote of McNamara's failings in Vietnam. Kill count was simply a terrible metric of victory. Vietnam was a civil war, which requires a political victory. This means allegiance of local leaders and populist support – both of which are measurable values. A kill count is often inversely correlated with political support. McNamara's failing here is more like the XY problem [1]. He incorrectly assumed that a brutal display of military power was the way to political victory and so he measured the wrong value. [1] - https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem

hristov 5y ago

A. Please do not center justify paragraphs, this is really annoying and makes things very hard to read. B. It is very tricky to make conclusions about life and business from US conquering military campaigns from within the US, because these campaigns have been shrouded with so many layers of lies self delusion and bullshit, that a lot of the generally accepted facts are just not true, and thus not able to teach us anything. By the way, I do not think America is in any way unique about this. Every country's conquering wars are carefully explained with heaps of bullshit from that country. There are some very rare exception as in with Germany and Japan and WW2, but usually every country carefully shapes the official story to make it so that the conquering wars are not completely based on greed and lust for power, when they usually are. So lets talk about McNamara and Vietnam. The problems there had little to do with some kind of duality between the quantitative and qualitative. In fact if McNamara honestly followed quantitative analysis, he would reach very different conclusions about Vietnam. If, for example, he commissioned a very simple very quantitative opinion poll among Vietnamese about whether they would prefer to be ruled by a puppet regime installed by a foreign country or by the independent patriotic leader that just defeated the French colonialists, he would see that the official reason for invading Vietnam was just not valid. It is not clear whether such a poll has been commissioned but McNamara probably knew what the results would be, because the US never permitted a free election in South Vietnam. Thus, faced with a simple quantitative fact against the war, the quantitative managerial McNamara ignored it and suppressed it. So no it is not about the quantitative vs the qualitative, it is the usual problem with wars of conquest by rich countries. People in positions that would benefit from the war carefully and persistently create a web of lies and delusions that is intended to drive the country to war.

dundarious 5y ago
I think the most HN-relevant elements of the fallacy are not about quantitative vs. qualitative. > The McNamara Fallacy is to presume that (A) quantitative models of reality are always more accurate than other models; (B) the quantitative measurements that can be made most easily must be the most relevant; and (C) factors other than those currently being used in quantitative metrics must either not exist or not have a significant influence on success. (A) is quant vs. qual, but (B) and (C) are about fooling yourself into thinking you're doing something useful just because you're quantitative. And while I agree with pretty much all your commentary about the war, I think this would be a very qualitative (not quantitative) assessment on McNamara's part: > It is not clear whether such a poll has been commissioned but McNamara probably knew what the results would be, because the US never permitted a free election in South Vietnam. Thus, faced with a simple quantitative fact against the war, the quantitative managerial McNamara ignored it and suppressed it.

morelandjs 5y ago

I don’t really understand the narrow definition of quantitative analysis referenced here and elsewhere. If you know your metrics are failing you, then you implicitly have a more advanced model of success. Define this more advanced mental model so it can be invalidated. If your mental model is not defined narrowly enough to be invalidated, then how can you improve upon it? I think the lesson here was that the measure of success was defined poorly, not that quantitative analysis is flawed a practice.

lvass 5y ago

I'm amazed they didn't mention McNamara's 100,000. There's so much to learn trying to not repeat this guy's mistakes. Project 100,000 is an enlightening piece of history, to say the least.

sloan 5y ago

“There’s the old apocryphal story that in 1967, they went to the basement of the Pentagon, when the mainframe computers took up the whole basement, and they put on the old punch cards everything you could quantify. Numbers of ships, numbers of tanks, numbers of helicopters, artillery, machine gun, ammo—everything you could quantify,” says James Willbanks, the chair of military history at U.S. Army Command and General Staff College. “They put it in the hopper and said, ‘When will we win in Vietnam?’ They went away on Friday and the thing ground away all weekend. [They] came back on Monday and there was one card in the output tray. And it said, 'You won in 1965.’” https://www.theatlantic.com/technology/archive/2017/10/the-computer-that-predicted-the-us-would-win-the-vietnam-war/542046/ https://www.theatlantic.com/technology/archive/2017/10/the-c...

craigmcnamara 5y ago

Tough to share a last name with this guy.

mhh__ 5y ago
Even stranger if it was the middle name too

_8j50 5y ago

The US did not lose the war against the VC, they stopped supporting south vietnam's war against the VC and withdrew because the war became unpopular. The distinction is important because ultimately it wasn't America's war and the VC did not overwhelm the US military into surrender. It was the civilian government of the US that directed an end of their involvement. If anything, China won a cold war against the US who feared a direct conflict and caved in. With Afghanistan, the war against the taliban government was won 15+ years ago. The US thought it could play nation-builder like with Japan. The continued involvement in afghanistan was part of the ongoing "war against terror" to reduce the chance of afghanistan becoming a heaven for terrorists again. The war against afghanistan was won, what the US lost was a prolonged occupation. If you look at history, the victor would more or less burn the invaded country to the ground and enslave its people to avoid similar long term issues, one can argue it was both self-interest and an ideology of being humane to civilians that resulted in the waste of post-war afghanistan occupation. Regarding metrics, I am fighting this right now. I found out what is expected of me early on and worked to extract that value out of myself and support my team but in the end my struggle is being able to explain metrics that reflect a years worth of work that only capture at best 30% of what is expected of me. Either way, can't complain, management isn't easy but meeting metrics is. what I dislike is if I have good metrics and stop there, it isn't enough. Metrics will always set a maximum ceiling on quality, if management can accept that I have no problem with metrics driven performance appraisal so long as it pays.

aiilns 5y ago
> The US thought it could play nation-builder like with Japan. The continued involvement in afghanistan was part of the ongoing "war against terror" to reduce the chance of afghanistan becoming a heaven for terrorists again. I would more easily characterize US external politics as nation-destroyer than nation-builder. Even if you disregard the evidence, it's quite a stretch to believe that the situation in Afghanistan is solely due to incompetence despite US good intentions. > If you look at history, the victor would more or less burn the invaded country to the ground and enslave its people to avoid similar long term issues Even if this has happened with some frequency I'm not so sure it's the rule; certainly there are many exceptions. In any case, it's no argument for doing the same.

toss1 5y ago

Drunk person, near streetlight, lost key ... Of course it is much easier to manage things that you can measure. That does NOT mean that if you cannot quantitatively measure it, that it can't be managed It certainly does not mean that if you haven't yet developed an effective metric, that it does not matter -- it's just mind-blowing that McNamara went that far. The way to do it is to FIRST figure out what matters. Then figure out if it can be measured, and if not, how to qualitatively assess it. Then also figure out how to ensure that the metrics don't become the goal (with perhaps the sole exception being if you are in sports and the goal is winning races/games, so that's a pretty good metric in a suite of metrics).