15 ms·

Be good-argument-driven, not data-driven

462 points by historynops 4y ago

deleted 4y ago
[deleted]
hackerlight 4y ago
> I originally claimed that data-driven culture leads bad arguments involving data to be favored over good arguments that don’t This is symptomatic of the deeper problem of thinking in terms of bumper stickers and slogans, instead of thinking from first principles. When it afflicts educated people, usually you hear slogans like "an anecdote is not data", or "that's the slippery slope fallacy". Instead of grappling with noisy reality, they have sharp cognitive categories with firm boundaries between concepts, then they try to squeeze things into these categories in order to make cognition easier because the relations between the categories are already understood. This gives them the illusion of rigorous and clear thought.
kqr 4y ago
While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data. On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning. (There's some, yes, but the main thrust of why I believe it is that variants of the experiment have been replicated reliably.) I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time. ---- Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise. ---- [1]: Here's just one relatively early example: http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/Dawes_Faust_Meehl_Clinical_vs_actuarial_assessments_1989.pdf http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/...
- uneoneuno 4y ago
  I feel like the author is leaning into comfort, intuitiveness. You bring up a fantastic point. Often we find data reveals things very unintuitive to human experience. We should always try to make Good Arguments - but without data they aren't always honest beyond feelings.
- zmgsabst 4y ago
  I find it strange that these are presented in tension, when they’re complementary. You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts. Deep understanding requires a model/narrative which fits the collection of data we have, and which allows us to reason about and predict the outcome of new situations. As Jeff Bezos put it: > Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design. > I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste. You won’t find any of it in a survey. https://www.aboutamazon.com/news/company-news/2016-letter-to-shareholders https://www.aboutamazon.com/news/company-news/2016-letter-to...
  - gchamonlive 4y ago
    I was about to write that in case of Bezos with Amazon, the customer was simpler and the answer was to just pour money into it until you substituted the market, but I realise now that that is not that simple. It seems simple because we have hindsight. My main idea though is that it is very hard to foresee what the customer will want after you deliver the product. Not what the customers want now, because sometimes they don't understand it until they experience it, and that makes me think that there is a LOT of luck at play here and a good deal of continency in prototype product design. Experience alone could be overrated. Think Kodak, I don't think they didn't have experience in product design, that they didn't understand their customers. I think they only didn't risk their luck and didn't think about what their customers would want in the future. And that is always a gamble. - Things are more nuanced and complex than I am putting it here, but bottom line is that I am trying to tap into survivors bias.
RandomLensman 4y ago
I often experience the inverse: people come up with hypotheses and theories that should see expressions in observable data - but no-one bothers to look and instead everyone argues around logical constructs etc.
quanto 4y ago
> A weak argument founded on poorly-interpreted data is not better than a well-reasoned argument founded on observation and theory. So a good argument is founded on...good data and good understanding of data? The article more seriously makes the mistake of begging the question: it presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments. But how do you know good arguments from bad arguments in the first place? What makes a good argument if not empirical data?
- gwd 4y ago
  > It presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments. It does indeed assume that there's a way to learn bad arguments from good; and so the focus should be on learning what are good argument and what are bad. > ...What makes a good argument if not empirical data? Consider the following conversation: A: We've done some numbers, and we've determined that there's a correlation between the number of firemen at a fire and the total damage done by the fire; with the fires handled by a single crew of three firemen doing the least damage. So we should limit all fire responses to a single crew to minimize damage. B: That doesn't make any sense -- of course we send more firemen to bigger fires, and bigger fires cause more destruction! If we take your advice, those big fires will cause even more damage! A: Hey, my argument is backed by empirical data; yours is just theoretical! Like, sure, it might be even better if B had empirical data to back him up; but even without that data, B should be winning the argument here. And the argument of the article is that many people espousing "data-driven" approaches end up being like A: Not scrutinizing the logic that they're using to analyze the data, and not acknowledging the limitations of what the data collected can say.
  - jjk166 4y ago
    You left out hypothesis C: if you send too many firefighters they get in eachothers way and become difficult to coordinate, making them worse at putting out the blaze. And hypothesis D: fires with the same number of firefighters cause different levels of destruction because some departments are organized to let their 10X firefighters work more efficiently. And hypothesis E: Many arsonists become firefighters thus more firefighters increases the risk that an arsonist will be on the team And hypothesis F: The same as hypothesis A but since some tools require more than one person there's actually a minimum threshold below which destruction skyrockets And hypothesis G: Wealthy areas that can hire more firefighters also suffer more expensive destruction for a given blaze. And hypothesis H: If we invest the resources we're spending on firefighters into fire prevention we can reduce total fire damage And infinitely more hypotheses. There will always be another argument that makes some logical sense. And unfortunately reality is under no obligation to make sense, so it's entirely possible something that sounds stupid and counterintuitive could just happen to be correct anyways. But with data, we can test hypotheses. Vary the number of firefighters and see what happens.
nordsieck 4y ago
One of the big reasons why data driven approaches are so seductive is, it's very difficult in the moment to distinguish between a good argument and a well crafted rationalization.
- gwd 4y ago
  The issue is that it doesn't fundamentally solve the problem. It's true that a good argument logically supported by data is better than a good argument that hasn't been checked against data. But the existence of data in the argument doesn't help you determine whether it's a good argument logically supported by data, or a well-crafted rationalization speciously supported by data.
crabmusket 4y ago
This reminds me a lot of the discussion of the scientific method by Karl Popper, and David Deutsch who was very influenced by Popper. "Being data-driven" sounds very empirical. Just look at the data, and see what you find in it. But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. Popper in Conjectures and Refutations: > Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems. Deutsch, in The Beginning of Infinity, emphasizes the importance of conjecture, and the role of observation as refuting or criticising those conjectures: > Where does [knowledge] come from? Empiricism said that we derive it from sensory experience. This is false. The real source of our theories is conjecture, and the real source of our knowledge is conjecture alternating with criticism. We create theories by rearranging, combining, altering and adding to existing ideas with the intention of improving upon them. The role of experiment and observation is to choose between existing theories, not to be the source of new ones. We interpret experiences through explanatory theories, but true explanations are not obvious. To bring this back to the subject of the article, I might suggest that it's possible to be "data driven" without a sound explanation or theory that the data is either interpreted through, or used to criticise. Or maybe such theories do exist, but are left implicit.
- JackFr 4y ago
  > But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. If you look at the heart attack data, and you ignore smoking you end up inventing the mythical Type A personality — but it was data driven. https://en.m.wikipedia.org/wiki/Type_A_and_Type_B_personality_theory https://en.m.wikipedia.org/wiki/Type_A_and_Type_B_personalit...
- guerrilla 4y ago
  > Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems. Thanks, I'd never heard this quote before. He's pretty much describing pragmatism à la William James. I had no idea.
marginalia_nu 4y ago
I think the problem is that people chronically underestimate how hard good science is. Professors get this wrong all the time, despite being some of the smartest people we have around, despite decades of experience and education, despite a career and reputation on the line, and despite a system of peer review to catch mistakes before they get published. Designing experiments is really difficult. Interpreting experiments is difficult and unintuitive. Statistics is difficult. You can't just look at whether the number went up. You need to have a deep understanding of significance, power and effect size, you should probably be doing ANOVA or some such.
stevejohnson 4y ago
[dead]
1-6 4y ago
“Resist! Be skeptical! Have no tolerance for poor arguments made with data. Keep intrinsic motivation alive.” the last sentence was the TL;DR
jasode 4y ago
To the author... I'd suggest a rewrite of what you're trying to communicate because your usage of "good-argument-driven" is a textbook example of Begging The Question: https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question For discussion's sake, let's go along with excluding data/metrics/science in pushing for arguments. In this framework, what exactly is a "good" argument based on? Gut feel? Opinion? There was a famous quote by Jim Barksdale, the former CEO of Netscape: "If we have data, let’s look at the data. If all we have are opinions, let’s go with mine." (So the tie-breaker in competing arguments in that case was "hierarchy-of-arguer-driven".) So Jane and Bob disagree on the next action to take. Jane thinks her argument is a "good argument" but has no data. But Bob thinks he has a "good argument" but no data. How does this thread's blog post help resolve the above scenario? (Blog's answer: you're driven by the one that has the good argument.) ... which is circular.
- deleted 4y ago
  [deleted]
- tdehnel 4y ago
  A simple answer to this is that good explanations are hard to vary. More here: https://www.lesswrong.com/posts/jcTsbaQ8hNc7qxwaQ/explanations-as-hard-to-vary-assertions https://www.lesswrong.com/posts/jcTsbaQ8hNc7qxwaQ/explanatio...
  - jasode 4y ago
    >A simple answer to this is that good explanations are hard to vary. But the "hard to vary" explanations were built up from observing data of smashing particles. E.g. from your link: - Frank Wilczek describes hard-to-vary-ness as follows "A theory begins to be perfect if any change makes it worse." He explains further using the Standard Model as an example of a hard-to-vary explanation: Too many gluons! But each of the eight colour gluons is there for a purpose. Together, they fulfil complete symmetry among the color charges. [...] No fudge factors or tweaks are available. This author's blog post about "data" also links to his previous post[1] about "science" leading one astray from "good arguments" is the opposite of "hard to vary" explanations. Here's the reason for the disconnect: The author is using the adjective "good" in his idiosyncratic way to describe the type of arguments that depend more on "storytelling" and "intrinsic motivation" -- rather than empirical science/data. Excerpt: - >And here is a secret: in the natural sciences themselves, storytelling and bare conjecture are far more important modes of persuasion than data-based empirical argument, anyway. [...] - >A good example of the sort of argument I think is helpful is A Philosophy of Software Design. Ousterhout defines his terms clearly, accompanies his definitions and claims with illustrative examples, and tells an occasional story. You, the reader, are free to evaluate each claim based on whether it plausibly seems to capture the essence of what you have encountered in your experiences writing software. For my part, I didn’t find most of Ousterhout’s ideas to be persuasive, as some of my colleagues did, but that doesn’t mean they aren’t good arguments, Those types of subjective claims arguments the author is espousing are actually "easy to vary" -- because they don't require constructing a cohesive theory that reconciles data that looks contradictory (e.g. like the The Standard Model, or Theory of General Relativity reconciling the speed-of-light observations). [1] http://twitchard.github.io/posts/2019-10-13-software-development-and-the-false-promise-of-science.html http://twitchard.github.io/posts/2019-10-13-software-develop...
thenerdhead 4y ago
To use Clayton Christensen’s theory of innovation here, to sustain innovation, businesses tend to be purely data driven. They continue to grow and make more money based on choices made with pure data. For disruptive innovation however, there needs to be an “argument” or opinion to help drive that data based on the industry trends. Companies then take a risk of delivering something new and good enough to the market. Also known as disruptive innovation. This has shifted the idea of being data-driven to being one of “data-inspired”. Anyone can make the same dataset fall into their favor. That’s the problem with being purely data-driven. Another way to think of it in the US especially is that our two party system makes wildly different conclusions from the same data. What’s preventing businesses from doing the same?
DoreenMichele 4y ago
See also the book How to lie with statistics and similar (I think a follow up book was called How to lie with charts and graphs).
contravariant 4y ago
The hidden assumption here is that things go well if and only if (you think) you understand all the factors that influence your metrics, can do experiments and are prepared to use fancy statistics. Which I reckon is a bit iffy. Special relativity was thought out well before any experiments to test it were feasible, and if understanding everything that influences your metric is a prerequisite then you can blame all failures on insufficient understanding without having any way of knowing when you have enough understanding.
quickthrower2 4y ago
"According to the data on business failures, you should have never started this business"
Shacklz 4y ago
> Are you prepared to do some very very fancy statistics? I'd extend this with "... while understanding what you're doing?" I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.
- blitzar 4y ago
  > Are you prepared to do some very very fancy statistics? IF you need 'fancy' statistics then it is not going to be a good data driven argument at all.
- bee_rider 4y ago
  I think "Be brutally honest about you many assumptions and caveats" at least implies that. I mean, in an informal setting there's room for an honest person to say "well I did some math and I don't really get it but I think it says...," but I think this article is addressed to software engineers and scientists. Someone representing themself as an engineer or scientists has a professional ethical responsibility to some sort of... I dunno, epistemic honesty, the knowledge of what their expertise covers, and communicating their limitations to laymen. The person with the A/B test in your example is either a liar because they are misrepresenting what their tool says, or they are a liar because they are misrepresenting their ability to tell you what it says, but either way they are a liar.
throwaway0asd 4y ago
A major exception to this reasoning is performance. Argument driven performance suggestions are wrong more than 80% of the time and likely wrong by several orders of magnitude. You can’t know just how wrong you are without appropriate data. This makes for a good litmus test of whether people are lying to you about software or, more likely, have absolutely no idea what they are doing.
- NateEag 4y ago
  Performance falls into the article's category of "things you can reliably measure." Thus, the author would agree that in performance optimization, you should collect and analyze data.
  - throwaway0asd 4y ago
    The problem isn’t what the article author believes, but rather what developers commonly (perhaps almost universally) believe. Most developers will fall back to intuition for any performance oriented decision even when they otherwise prefer data oriented decisions and even when the task at hand is critical to the health of their product/business. This is because performance measures require: 1. Additional effort 2. (most importantly) A willingness to abandon familiar concepts of approach Sometimes such decisions vested in intuition are truth by omission, a form of lying, because the resulting self-comfort is worth more than the numeric benefits.
jason-phillips 4y ago
This reminds me of the Principal Chalmers meme. In this case, first pondering whether he is wrong, only to conclude that it's the data that's wrong. I know that's not what the article says per se, but it's only one slightly abstracted reinterpretation removed, as OP's title demonstrates.
- zmgsabst 4y ago
  Minor nit: Principal Skinner; Chalmers was the superintendent. https://www.knowyourmeme.com/memes/am-i-so-out-of-touch https://www.knowyourmeme.com/memes/am-i-so-out-of-touch
  - xdavidliu 4y ago
    good point. Still; I would've presumed Chalmers was superintendent at some point in his career. Additionally, Chalmers has on occasion [1] been referred to as "Super Nintendo Chalmers". [1] https://www.youtube.com/watch?v=av4lbel9aIo https://www.youtube.com/watch?v=av4lbel9aIo
  - jason-phillips 4y ago
    Doh!
moralestapia 4y ago
Sure, but the thing with "good arguments" is that when two hypotheses oppose each other, it is the case that supporters on each side are sure they are behind the "good argument" so ... Data doesn't lie; it could be nuanced, yes, but if its truthful then you cannot really argue against that.
allsunny 4y ago
I won’t belabor the point because others have already made it: this article assumes there is some way to sort through good and bad arguments in the absence of data - a pretty big leap. The reality is all of our arguments are appealing to some sort of data (eg previous experience), it’s just that it doesn’t always fit in a neat definition of data. Obligatory: https://en.m.wikipedia.org/wiki/All_models_are_wrong https://en.m.wikipedia.org/wiki/All_models_are_wrong
- ajkjk 4y ago
  "Previous experience" is not what is meant by 'data' in this industry. If company's decision-making was including both data and experience/wisdom/intuition, it wouldn't be so frustratingly wrong all the time.
  - allsunny 4y ago
    I agree that's not what is meant by 'data' in the industry and I'm challenging that a little bit. However, even if we use the industry definition, what you're saying is hyperbole. Every company uses both data and experience to varying degrees. People get hung up when they think the balance isn't appropriate - not surprisingly, that happens when one or the other doesn't support their opinion. I'd rather be in a position of defending my opinion with data. It's already been quoted but... "If we have data, let's look at data. If all we have are opinions, let's go with mine."
- HPsquared 4y ago
  There's lies, damn lies, and statistics. Models are further along, beyond statistics.
  - allsunny 4y ago
    Models are just applied statistics?
shubb 4y ago
The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basic statistical tests. Especially in the B2C space a lot of companies, departments, products don't actually have a lot of customers and certainly not many decision makers. The N number is always going to be low. You can just talk to people. Let's say you are doing pretty well and running a SaS with 1000 corporate customers paying a million each - that's a billion dollar revenue - you can just talk to them. Certainly you can just talk to every single person who signs the cheque and those are the only people that matter. And which is easier - putting together a thorough suite of A/B tests or getting some real customers to use your app on video and talking to them about what they are finding annoying, useful, missing? I see less people do that than you'd think.
- bell-cot 4y ago
  Unfortunately, your reality-driven approach has ~zero emotional appeal for most managers, exec's, and alpha-data-scientist wanna-be's.
  - germinalphrase 4y ago
    Why? Inadequately “technical”?
    - jkingsbery 4y ago
      I think there are lots of reasons why. One possible reason: no one whose job it is to write Python scripts was ever promoted for making an Excel spreadsheet when that is the simpler and more practical approach. And no manager of people who write Python scripts is going to be able to use that Excel spreadsheet to sell "I need more responsibility and head count." People tend to follow incentives, rather than focusing on making wise decisions.
oxfordmale 4y ago
This is not what the data shows https://www.google.com/search?q=data+driven+companies+more+profitable&oq=data+driven+companies+more+profitable&aqs=chrome..69i57j0i546j0i30i546j0i546l3.4938j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=data+driven+companies+more+p... Any good-argument-driven based argument you attempt to make is almost always based on political motivating factors, rather on what is good for the business. Intuition driven decisions work when the market is behaving normally, however, are generally too slow in a fast changing market like we have been since the start of COVID.
- tdehnel 4y ago
  > Any good-argument-driven based argument you attempt to make is almost always based on political motivating factors If this is true in the case of a specific theory, then that is not a good theory.
  - oxfordmale 4y ago
    I was mostly referring to business decisions. For that type of decisions there are always political factors at play (building empires, career growth, dislike for another person/team) that do not necessarily align with business success. Lehman Brothers is one of those examples.
    - tdehnel 4y ago
      You’re actually proving my point. Those so-called successful business decisions never turn out to be true enough to work long term. Like all knowledge, they are eventually shown to be false.
      - oxfordmale 4y ago
        Data driven companies, like Amazon, tend to do better, on average than other companies. However, this doesn't mean they are immune to mistakes. At some point in time Amazon wil fall in decline and disappear, however, that doesn't imply that being data driven was a bad decision.
ekianjo 4y ago
Good argument is just another name for confirmation bias, most of the time.
viridian 4y ago
This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy. In this case the topic of value is the often fraught relationship between empiricism and rationalism, and the impacts each have on the scientific process, research, education, and how we go about understanding the world. To operate with one with a complete absence of the other is to expose yourself to huge, often fundamental gaps in your thinking, your arguments, and your plans. This is what the author is ultimately getting at from the direction of the empirical: data, in the form of a large collection of discrete observations, can be used to justify a sea of mutually exclusive claims that may or may not be in accordance with reality, and that's to say nothing about the quality of the data itself.
- dalbasal 4y ago
  Idk how if studying philosophy helps. Most philosophers were/are themselves committed to one school or theory, with gaps galore. In any case, I think empirical science's defeat of rationalism ( eg Galileo Vs Church) has all sorry of ramifications. Social sciences like economics and psychology have a lot of trouble bridging the gaps.
  - polio 4y ago
    Epistemology is a subfield of philosophy. Seems like a healthy understanding of that would be good for society right now. > Most philosophers were/are themselves committed to one school or theory, with gaps galore. Most scientists specialize one thing, but students of science don't. One can learn about many schools of philosophy, as well.
- eufyvodsk 4y ago
  The problem with this is that philosophy isn't a magical panacea that illuminates the way towards a more ideal state. It can be used to justify a sea of mutually exclusive claims that may not be in accordance with reality, and that's to say nothing about the quality of the arguments themselves.
- verisimi 4y ago
  I agree so strongly with this. The point I would add is that hardly anyone uses the empirical process directly. It is all 'this article claims this' or 'this study says that'. It's very 'meta' with little to no personal verification or testing of the claims - ie, theories based on theories or models based on models, or maps based on maps. Very few check the terrain itself to confirm that the map applies. We trust education, experts, peer review etc. We're drowning in models, especially as these are easily represented on computers, but have no ability to check the models against reality. PS this disassociation from reality will not improve as we move forward technologically. No doubt, in the metaverse we will be able to create ever more elaborate models, or is it that we will be ever more disassociated from our own anecdotal experiences? (Where 'anecdotal' is something to apologise about).
apienx 4y ago
Being data-driven for the sake is being data-driven is indeed becoming an issue. The resources spent measuring and analysing data are overwhelmingly larger than they should in most cases. Cohorts of "data scientists" and "managers" dive head on into data without much (if any!) first-principles thinking. People tend to replicate metrics without much thought into their relevance to the specific situation. Thinking properly is a very hard skill to acquire (the hardest?), and most do everything they can to avoid it. "What you measure affects what you do. If you don't measure the right thing, you don't do the right thing." -- Joseph Stiglitz
ThomPete 4y ago
David Deutsch (Father of Quantum Computation and one of the most brilliant human beings alive) have a really great way of thinking these kinds of discussions. He calls it good explanations. A good explanation is something that is hard to vary while still solving the problem it purports to solve. He is against most use of Bayesianism when used for predictions. Great presentation here https://www.youtube.com/watch?v=EVwjofV5TgU https://www.youtube.com/watch?v=EVwjofV5TgU
tgtweak 4y ago
I've seen a lot of good arguments put to rest with a good test. The key is collecting and looking at the data correctly. Data without a keen understanding of why you need it and what you're looking to solve with it is not much use.
Alex3917 4y ago
> Be good-argument-driven, not data-driven FWIW the proper term is "data-informed."
yarosh 4y ago
1. If there are no good arguments in the collective - there's no retrospective and it's primarily a management and psychological issue. No one is able to fully self-reflect and it breaks the existing delegation / escalation chains, respectively. 2. If there are no viable data sources, when it can be proven that there's a correlation with an actual business processes, - it's a management problem. People Can't establish viable metrics, once again, mostly due to 1. This is something any company of any size and any budget can struggle with due to lack of XP and the usual collective XP-accumulation / knowledge sharing deficiency. You can't self-reflect onto something you haven't learned about, yet. And due to 1 this is a closed loop because lack of XP can't be escalated accordingly, most of the time it's also a Workplace Deviance factor. 3. Practically, it ends up in a bouquet of Workplace Deviance because no one in the end will be willing to take the blame and actual responsibility to fix anything. Any Problem vs Solution type of culture will worsen things a lot i.e. "All the blame and no Compassion". Companies are usually forced to adopt some Teal stuff in the end, maybe for really no other good reason, but just to keep on growing. The idea of hiring HR that can "work by the booK" and actually build up a personal profile of how anyone could fit into all this mess is impossible by definition - due to Employee Silence and broken retro no one will be willing to expose all the shit that is happening, in the first place... So, most of the time I see Kitchen Sink companies with volatile outcomes where there really no one who could even be able to listen to any arguments, in the first place. Google's internal ML-driven productivity metrics became a meme already for all the reasons described above. You can't reason with Toxic and Inadequate people. Also Asana claim that Social Loafing is a myth and everything else is a retro deficiency really wrong - retro can prevent and display certain glorious occasions, but it's not a root cause of any psychological effect by definition.
stuckinhell 4y ago
Be politically driven (company politics) driven. Good arguments should take in account people's ambitions, and political aspirations especially at big fortune 500 companies. Startups can be more honest.
SpicyLemonZest 4y ago
Great article, but I think it somewhat misunderstands the impetus for the concept. "Data has its place" sounds obvious precisely because "data-driven" has been such a successful concept. The alternative perspective, which used to be very common in our industry and still pops up from time to time, is that metrics are something you write for debugging and business decisions are made by gut feeling or abstract philosophical analysis. (Most software companies had to make decisions this way in the pre-cloud era, because it wasn't usually feasible to collect usage metrics.)
cptcobalt 4y ago
I couldn't agree with this more. I feel like the author took some of the arguments straight from my brain—I'm exhausted by pseudoscientific "data-driven" arguments. From my experience, most of these try to distill an incredibly complex problem space down to a one-dimensional black and white decision. But the real world doesn't work like that–it's full of grey area, and things we can't effectively measure. If you're trying to slice and dice data down to a happy one-dimensional decision point, you're often missing or ignoring important detail. At work, I'm far more happy with postmortems with general, open "good/bad" lists of after the fact feedback, that we use to consider how we prioritize and design what comes next.
ifsothen 4y ago
Yes, data is useless without a qualitative explanation. There are simply too many possible confounding factors that you cannot eliminate without understanding what they may be.
ltbarcly3 4y ago
Being argument driven gives control to the organization's 'lawyers'. People can be very persuasive independent of the reality of the situation.
N1H1L 4y ago
Be data-driven, and question the provenance of your data all the time. Otherwise you will end up like economics, a field with prettier models and more mathematics than almost every engineering field, and yet gets every major prediction wrong.
tanvach 4y ago
I’ll probably be buried in all these comments, but my position is that data is only as good as how it is collected. Sloppy data collection gives rise to sloppy conclusion through unknown biases. The key is to understand the ‘data generation process’ so you can identify biases. My experience suggests that doing so side-step some common pitfalls. I recommend reach out for ‘The Book Of Why’ by Judea Pearl. He includes many real life examples that’s surprisingly applicable to modern data science.
UIUC_06 4y ago
Good article. When your only tool is a hammer, every problem looks like a thumb. While we're at it: I've actually been in scrums where the "burndown rate" was analyzed as if it was actually A Thing. It is not A Thing.
colo_innerself 4y ago
A whole book was written on this very topic: "The Tyranny of Metrics" by Jerry Z. Muller https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metrics https://press.princeton.edu/books/hardcover/9780691174952/th...
romankolpak 4y ago
I have experienced this first hand, so this article resonates a lot with me. I worked with a manager who prioritized work which was easily measurable, so he could report the good numbers to leadership and get career points out of this. Unfortunately the project we took on was a demanding and technically challenging problem, and in almost a year of work of a team of engineers we made barely any real progress or made any actual difference, but the numbers were great and people were satisfied during presentations. I ended up feeling completely disconnected from my job and losing all motivation to work there.
taeric 4y ago
I don't know. There are a ton of great arguments that will lead to dead ends and stalled projects. :(
dkbrk 4y ago
I'm surprised there's no mention of Goodhart's Law [0]. Even if the metric is "well understood and free from human/social factors", once you start using it as a target that will no longer be the case. [0]: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
borski 4y ago
In Range, David Epstein talks about about NASA and some of their disasters, like the explosion of Challenger. NASA is the entirely encased in specialized knowledge, and has a completely data-driven mindset, with no room for logic. If you can't prove it with data, they wouldn't even consider it. He explains that, “Reason without numbers was not accepted. In the face of an unfamiliar challenge, NASA managers failed to drop their familiar tools... The Challenger managers made mistakes of conformity. They stuck to the usual tools in the face of an unusual challenge.” Even though the mistake that led to the Challenger disaster could have been caught, it was the uniformity of thinking that lead to an organizational blind spot, and that uniformity was to be too focused on data-driven arguments. There is a famous call prior to the disaster on which engineers had raised the concerns but it was based on intuition and a few cherry picked samples, not a full set of data, and this was the night before the launch. Because of the lack of data, they went ahead with it and we all know the tragedy that ensued. Moreover, other engineers who agreed that there was an issue didn't speak up, because they too lacked the data, and knew that management wouldn't care.
blueyes 4y ago
Key idea is the "data maturity" of the topic under discussion. Where there is data, you should use it and be smart about it. For a lot of big decisions, especially in companies doing something new, there is no good data at first. You have to reason about it based on experience and analogy. Then, once you commit to a path, you can start gathering data to see if your hypothesis was correct. The further you go, the more you can rely on data, assuming you know how to think about it. Discussions about being data-driven that don't take into account the "data maturity" of the situation are nonsensical. Being "data driven" when you're considering something radically new is either delusional or a cop out. Ignoring data when it could correct your biases is either lazy or wrong or both. And finally, lots of people who claim to be "data driven" are not smart about data. To paraphrase Wilde, "data is rarely pure and never simple." It doesn't just reveal truths you can treat as dogma. It's ambiguous and takes a lot of work to interpret. A lot of "data driven" teams aren't doing that work.

462 points by historynops 4y ago

deleted 4y ago

[deleted]

> I originally claimed that data-driven culture leads bad arguments involving data to be favored over good arguments that don’t This is symptomatic of the deeper problem of thinking in terms of bumper stickers and slogans, instead of thinking from first principles. When it afflicts educated people, usually you hear slogans like "an anecdote is not data", or "that's the slippery slope fallacy". Instead of grappling with noisy reality, they have sharp cognitive categories with firm boundaries between concepts, then they try to squeeze things into these categories in order to make cognition easier because the relations between the categories are already understood. This gives them the illusion of rigorous and clear thought.

kqr 4y ago

While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data. On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning. (There's some, yes, but the main thrust of why I believe it is that variants of the experiment have been replicated reliably.) I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time. ---- Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise. ---- [1]: Here's just one relatively early example: http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/Dawes_Faust_Meehl_Clinical_vs_actuarial_assessments_1989.pdf http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/...

uneoneuno 4y ago
I feel like the author is leaning into comfort, intuitiveness. You bring up a fantastic point. Often we find data reveals things very unintuitive to human experience. We should always try to make Good Arguments - but without data they aren't always honest beyond feelings.
zmgsabst 4y ago
I find it strange that these are presented in tension, when they’re complementary. You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts. Deep understanding requires a model/narrative which fits the collection of data we have, and which allows us to reason about and predict the outcome of new situations. As Jeff Bezos put it: > Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design. > I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste. You won’t find any of it in a survey. https://www.aboutamazon.com/news/company-news/2016-letter-to-shareholders https://www.aboutamazon.com/news/company-news/2016-letter-to...
- gchamonlive 4y ago
  I was about to write that in case of Bezos with Amazon, the customer was simpler and the answer was to just pour money into it until you substituted the market, but I realise now that that is not that simple. It seems simple because we have hindsight. My main idea though is that it is very hard to foresee what the customer will want after you deliver the product. Not what the customers want now, because sometimes they don't understand it until they experience it, and that makes me think that there is a LOT of luck at play here and a good deal of continency in prototype product design. Experience alone could be overrated. Think Kodak, I don't think they didn't have experience in product design, that they didn't understand their customers. I think they only didn't risk their luck and didn't think about what their customers would want in the future. And that is always a gamble. - Things are more nuanced and complex than I am putting it here, but bottom line is that I am trying to tap into survivors bias.

RandomLensman 4y ago

I often experience the inverse: people come up with hypotheses and theories that should see expressions in observable data - but no-one bothers to look and instead everyone argues around logical constructs etc.

quanto 4y ago

> A weak argument founded on poorly-interpreted data is not better than a well-reasoned argument founded on observation and theory. So a good argument is founded on...good data and good understanding of data? The article more seriously makes the mistake of begging the question: it presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments. But how do you know good arguments from bad arguments in the first place? What makes a good argument if not empirical data?

gwd 4y ago
> It presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments. It does indeed assume that there's a way to learn bad arguments from good; and so the focus should be on learning what are good argument and what are bad. > ...What makes a good argument if not empirical data? Consider the following conversation: A: We've done some numbers, and we've determined that there's a correlation between the number of firemen at a fire and the total damage done by the fire; with the fires handled by a single crew of three firemen doing the least damage. So we should limit all fire responses to a single crew to minimize damage. B: That doesn't make any sense -- of course we send more firemen to bigger fires, and bigger fires cause more destruction! If we take your advice, those big fires will cause even more damage! A: Hey, my argument is backed by empirical data; yours is just theoretical! Like, sure, it might be even better if B had empirical data to back him up; but even without that data, B should be winning the argument here. And the argument of the article is that many people espousing "data-driven" approaches end up being like A: Not scrutinizing the logic that they're using to analyze the data, and not acknowledging the limitations of what the data collected can say.
- jjk166 4y ago
  You left out hypothesis C: if you send too many firefighters they get in eachothers way and become difficult to coordinate, making them worse at putting out the blaze. And hypothesis D: fires with the same number of firefighters cause different levels of destruction because some departments are organized to let their 10X firefighters work more efficiently. And hypothesis E: Many arsonists become firefighters thus more firefighters increases the risk that an arsonist will be on the team And hypothesis F: The same as hypothesis A but since some tools require more than one person there's actually a minimum threshold below which destruction skyrockets And hypothesis G: Wealthy areas that can hire more firefighters also suffer more expensive destruction for a given blaze. And hypothesis H: If we invest the resources we're spending on firefighters into fire prevention we can reduce total fire damage And infinitely more hypotheses. There will always be another argument that makes some logical sense. And unfortunately reality is under no obligation to make sense, so it's entirely possible something that sounds stupid and counterintuitive could just happen to be correct anyways. But with data, we can test hypotheses. Vary the number of firefighters and see what happens.

nordsieck 4y ago

One of the big reasons why data driven approaches are so seductive is, it's very difficult in the moment to distinguish between a good argument and a well crafted rationalization.

gwd 4y ago
The issue is that it doesn't fundamentally solve the problem. It's true that a good argument logically supported by data is better than a good argument that hasn't been checked against data. But the existence of data in the argument doesn't help you determine whether it's a good argument logically supported by data, or a well-crafted rationalization speciously supported by data.

crabmusket 4y ago

This reminds me a lot of the discussion of the scientific method by Karl Popper, and David Deutsch who was very influenced by Popper. "Being data-driven" sounds very empirical. Just look at the data, and see what you find in it. But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. Popper in Conjectures and Refutations: > Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems. Deutsch, in The Beginning of Infinity, emphasizes the importance of conjecture, and the role of observation as refuting or criticising those conjectures: > Where does [knowledge] come from? Empiricism said that we derive it from sensory experience. This is false. The real source of our theories is conjecture, and the real source of our knowledge is conjecture alternating with criticism. We create theories by rearranging, combining, altering and adding to existing ideas with the intention of improving upon them. The role of experiment and observation is to choose between existing theories, not to be the source of new ones. We interpret experiences through explanatory theories, but true explanations are not obvious. To bring this back to the subject of the article, I might suggest that it's possible to be "data driven" without a sound explanation or theory that the data is either interpreted through, or used to criticise. Or maybe such theories do exist, but are left implicit.

JackFr 4y ago
> But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. If you look at the heart attack data, and you ignore smoking you end up inventing the mythical Type A personality — but it was data driven. https://en.m.wikipedia.org/wiki/Type_A_and_Type_B_personality_theory https://en.m.wikipedia.org/wiki/Type_A_and_Type_B_personalit...
guerrilla 4y ago
> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems. Thanks, I'd never heard this quote before. He's pretty much describing pragmatism à la William James. I had no idea.

marginalia_nu 4y ago

I think the problem is that people chronically underestimate how hard good science is. Professors get this wrong all the time, despite being some of the smartest people we have around, despite decades of experience and education, despite a career and reputation on the line, and despite a system of peer review to catch mistakes before they get published. Designing experiments is really difficult. Interpreting experiments is difficult and unintuitive. Statistics is difficult. You can't just look at whether the number went up. You need to have a deep understanding of significance, power and effect size, you should probably be doing ANOVA or some such.

stevejohnson 4y ago

[dead]

1-6 4y ago

“Resist! Be skeptical! Have no tolerance for poor arguments made with data. Keep intrinsic motivation alive.” the last sentence was the TL;DR

jasode 4y ago

To the author... I'd suggest a rewrite of what you're trying to communicate because your usage of "good-argument-driven" is a textbook example of Begging The Question: https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question For discussion's sake, let's go along with excluding data/metrics/science in pushing for arguments. In this framework, what exactly is a "good" argument based on? Gut feel? Opinion? There was a famous quote by Jim Barksdale, the former CEO of Netscape: "If we have data, let’s look at the data. If all we have are opinions, let’s go with mine." (So the tie-breaker in competing arguments in that case was "hierarchy-of-arguer-driven".) So Jane and Bob disagree on the next action to take. Jane thinks her argument is a "good argument" but has no data. But Bob thinks he has a "good argument" but no data. How does this thread's blog post help resolve the above scenario? (Blog's answer: you're driven by the one that has the good argument.) ... which is circular.

deleted 4y ago
[deleted]
tdehnel 4y ago
A simple answer to this is that good explanations are hard to vary. More here: https://www.lesswrong.com/posts/jcTsbaQ8hNc7qxwaQ/explanations-as-hard-to-vary-assertions https://www.lesswrong.com/posts/jcTsbaQ8hNc7qxwaQ/explanatio...
- jasode 4y ago
  >A simple answer to this is that good explanations are hard to vary. But the "hard to vary" explanations were built up from observing data of smashing particles. E.g. from your link: - Frank Wilczek describes hard-to-vary-ness as follows "A theory begins to be perfect if any change makes it worse." He explains further using the Standard Model as an example of a hard-to-vary explanation: Too many gluons! But each of the eight colour gluons is there for a purpose. Together, they fulfil complete symmetry among the color charges. [...] No fudge factors or tweaks are available. This author's blog post about "data" also links to his previous post[1] about "science" leading one astray from "good arguments" is the opposite of "hard to vary" explanations. Here's the reason for the disconnect: The author is using the adjective "good" in his idiosyncratic way to describe the type of arguments that depend more on "storytelling" and "intrinsic motivation" -- rather than empirical science/data. Excerpt: - >And here is a secret: in the natural sciences themselves, storytelling and bare conjecture are far more important modes of persuasion than data-based empirical argument, anyway. [...] - >A good example of the sort of argument I think is helpful is A Philosophy of Software Design. Ousterhout defines his terms clearly, accompanies his definitions and claims with illustrative examples, and tells an occasional story. You, the reader, are free to evaluate each claim based on whether it plausibly seems to capture the essence of what you have encountered in your experiences writing software. For my part, I didn’t find most of Ousterhout’s ideas to be persuasive, as some of my colleagues did, but that doesn’t mean they aren’t good arguments, Those types of subjective claims arguments the author is espousing are actually "easy to vary" -- because they don't require constructing a cohesive theory that reconciles data that looks contradictory (e.g. like the The Standard Model, or Theory of General Relativity reconciling the speed-of-light observations). [1] http://twitchard.github.io/posts/2019-10-13-software-development-and-the-false-promise-of-science.html http://twitchard.github.io/posts/2019-10-13-software-develop...

thenerdhead 4y ago

To use Clayton Christensen’s theory of innovation here, to sustain innovation, businesses tend to be purely data driven. They continue to grow and make more money based on choices made with pure data. For disruptive innovation however, there needs to be an “argument” or opinion to help drive that data based on the industry trends. Companies then take a risk of delivering something new and good enough to the market. Also known as disruptive innovation. This has shifted the idea of being data-driven to being one of “data-inspired”. Anyone can make the same dataset fall into their favor. That’s the problem with being purely data-driven. Another way to think of it in the US especially is that our two party system makes wildly different conclusions from the same data. What’s preventing businesses from doing the same?

DoreenMichele 4y ago

See also the book How to lie with statistics and similar (I think a follow up book was called How to lie with charts and graphs).

contravariant 4y ago

The hidden assumption here is that things go well if and only if (you think) you understand all the factors that influence your metrics, can do experiments and are prepared to use fancy statistics. Which I reckon is a bit iffy. Special relativity was thought out well before any experiments to test it were feasible, and if understanding everything that influences your metric is a prerequisite then you can blame all failures on insufficient understanding without having any way of knowing when you have enough understanding.

quickthrower2 4y ago

"According to the data on business failures, you should have never started this business"

Shacklz 4y ago

> Are you prepared to do some very very fancy statistics? I'd extend this with "... while understanding what you're doing?" I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.

blitzar 4y ago
> Are you prepared to do some very very fancy statistics? IF you need 'fancy' statistics then it is not going to be a good data driven argument at all.
bee_rider 4y ago
I think "Be brutally honest about you many assumptions and caveats" at least implies that. I mean, in an informal setting there's room for an honest person to say "well I did some math and I don't really get it but I think it says...," but I think this article is addressed to software engineers and scientists. Someone representing themself as an engineer or scientists has a professional ethical responsibility to some sort of... I dunno, epistemic honesty, the knowledge of what their expertise covers, and communicating their limitations to laymen. The person with the A/B test in your example is either a liar because they are misrepresenting what their tool says, or they are a liar because they are misrepresenting their ability to tell you what it says, but either way they are a liar.

throwaway0asd 4y ago

A major exception to this reasoning is performance. Argument driven performance suggestions are wrong more than 80% of the time and likely wrong by several orders of magnitude. You can’t know just how wrong you are without appropriate data. This makes for a good litmus test of whether people are lying to you about software or, more likely, have absolutely no idea what they are doing.

NateEag 4y ago
Performance falls into the article's category of "things you can reliably measure." Thus, the author would agree that in performance optimization, you should collect and analyze data.
- throwaway0asd 4y ago
  The problem isn’t what the article author believes, but rather what developers commonly (perhaps almost universally) believe. Most developers will fall back to intuition for any performance oriented decision even when they otherwise prefer data oriented decisions and even when the task at hand is critical to the health of their product/business. This is because performance measures require: 1. Additional effort 2. (most importantly) A willingness to abandon familiar concepts of approach Sometimes such decisions vested in intuition are truth by omission, a form of lying, because the resulting self-comfort is worth more than the numeric benefits.

jason-phillips 4y ago

This reminds me of the Principal Chalmers meme. In this case, first pondering whether he is wrong, only to conclude that it's the data that's wrong. I know that's not what the article says per se, but it's only one slightly abstracted reinterpretation removed, as OP's title demonstrates.

zmgsabst 4y ago
Minor nit: Principal Skinner; Chalmers was the superintendent. https://www.knowyourmeme.com/memes/am-i-so-out-of-touch https://www.knowyourmeme.com/memes/am-i-so-out-of-touch
- xdavidliu 4y ago
  good point. Still; I would've presumed Chalmers was superintendent at some point in his career. Additionally, Chalmers has on occasion [1] been referred to as "Super Nintendo Chalmers". [1] https://www.youtube.com/watch?v=av4lbel9aIo https://www.youtube.com/watch?v=av4lbel9aIo
- jason-phillips 4y ago
  Doh!

moralestapia 4y ago

Sure, but the thing with "good arguments" is that when two hypotheses oppose each other, it is the case that supporters on each side are sure they are behind the "good argument" so ... Data doesn't lie; it could be nuanced, yes, but if its truthful then you cannot really argue against that.

allsunny 4y ago

I won’t belabor the point because others have already made it: this article assumes there is some way to sort through good and bad arguments in the absence of data - a pretty big leap. The reality is all of our arguments are appealing to some sort of data (eg previous experience), it’s just that it doesn’t always fit in a neat definition of data. Obligatory: https://en.m.wikipedia.org/wiki/All_models_are_wrong https://en.m.wikipedia.org/wiki/All_models_are_wrong

ajkjk 4y ago
"Previous experience" is not what is meant by 'data' in this industry. If company's decision-making was including both data and experience/wisdom/intuition, it wouldn't be so frustratingly wrong all the time.
- allsunny 4y ago
  I agree that's not what is meant by 'data' in the industry and I'm challenging that a little bit. However, even if we use the industry definition, what you're saying is hyperbole. Every company uses both data and experience to varying degrees. People get hung up when they think the balance isn't appropriate - not surprisingly, that happens when one or the other doesn't support their opinion. I'd rather be in a position of defending my opinion with data. It's already been quoted but... "If we have data, let's look at data. If all we have are opinions, let's go with mine."
HPsquared 4y ago
There's lies, damn lies, and statistics. Models are further along, beyond statistics.
- allsunny 4y ago
  Models are just applied statistics?

shubb 4y ago

The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basic statistical tests. Especially in the B2C space a lot of companies, departments, products don't actually have a lot of customers and certainly not many decision makers. The N number is always going to be low. You can just talk to people. Let's say you are doing pretty well and running a SaS with 1000 corporate customers paying a million each - that's a billion dollar revenue - you can just talk to them. Certainly you can just talk to every single person who signs the cheque and those are the only people that matter. And which is easier - putting together a thorough suite of A/B tests or getting some real customers to use your app on video and talking to them about what they are finding annoying, useful, missing? I see less people do that than you'd think.

bell-cot 4y ago
Unfortunately, your reality-driven approach has ~zero emotional appeal for most managers, exec's, and alpha-data-scientist wanna-be's.
- germinalphrase 4y ago
  Why? Inadequately “technical”?
  - jkingsbery 4y ago
    I think there are lots of reasons why. One possible reason: no one whose job it is to write Python scripts was ever promoted for making an Excel spreadsheet when that is the simpler and more practical approach. And no manager of people who write Python scripts is going to be able to use that Excel spreadsheet to sell "I need more responsibility and head count." People tend to follow incentives, rather than focusing on making wise decisions.

oxfordmale 4y ago

This is not what the data shows https://www.google.com/search?q=data+driven+companies+more+profitable&oq=data+driven+companies+more+profitable&aqs=chrome..69i57j0i546j0i30i546j0i546l3.4938j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=data+driven+companies+more+p... Any good-argument-driven based argument you attempt to make is almost always based on political motivating factors, rather on what is good for the business. Intuition driven decisions work when the market is behaving normally, however, are generally too slow in a fast changing market like we have been since the start of COVID.

tdehnel 4y ago
> Any good-argument-driven based argument you attempt to make is almost always based on political motivating factors If this is true in the case of a specific theory, then that is not a good theory.
- oxfordmale 4y ago
  I was mostly referring to business decisions. For that type of decisions there are always political factors at play (building empires, career growth, dislike for another person/team) that do not necessarily align with business success. Lehman Brothers is one of those examples.
  - tdehnel 4y ago
    You’re actually proving my point. Those so-called successful business decisions never turn out to be true enough to work long term. Like all knowledge, they are eventually shown to be false.
    - oxfordmale 4y ago
      Data driven companies, like Amazon, tend to do better, on average than other companies. However, this doesn't mean they are immune to mistakes. At some point in time Amazon wil fall in decline and disappear, however, that doesn't imply that being data driven was a bad decision.

ekianjo 4y ago

Good argument is just another name for confirmation bias, most of the time.

viridian 4y ago

This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy. In this case the topic of value is the often fraught relationship between empiricism and rationalism, and the impacts each have on the scientific process, research, education, and how we go about understanding the world. To operate with one with a complete absence of the other is to expose yourself to huge, often fundamental gaps in your thinking, your arguments, and your plans. This is what the author is ultimately getting at from the direction of the empirical: data, in the form of a large collection of discrete observations, can be used to justify a sea of mutually exclusive claims that may or may not be in accordance with reality, and that's to say nothing about the quality of the data itself.

dalbasal 4y ago
Idk how if studying philosophy helps. Most philosophers were/are themselves committed to one school or theory, with gaps galore. In any case, I think empirical science's defeat of rationalism ( eg Galileo Vs Church) has all sorry of ramifications. Social sciences like economics and psychology have a lot of trouble bridging the gaps.
- polio 4y ago
  Epistemology is a subfield of philosophy. Seems like a healthy understanding of that would be good for society right now. > Most philosophers were/are themselves committed to one school or theory, with gaps galore. Most scientists specialize one thing, but students of science don't. One can learn about many schools of philosophy, as well.
eufyvodsk 4y ago
The problem with this is that philosophy isn't a magical panacea that illuminates the way towards a more ideal state. It can be used to justify a sea of mutually exclusive claims that may not be in accordance with reality, and that's to say nothing about the quality of the arguments themselves.
verisimi 4y ago
I agree so strongly with this. The point I would add is that hardly anyone uses the empirical process directly. It is all 'this article claims this' or 'this study says that'. It's very 'meta' with little to no personal verification or testing of the claims - ie, theories based on theories or models based on models, or maps based on maps. Very few check the terrain itself to confirm that the map applies. We trust education, experts, peer review etc. We're drowning in models, especially as these are easily represented on computers, but have no ability to check the models against reality. PS this disassociation from reality will not improve as we move forward technologically. No doubt, in the metaverse we will be able to create ever more elaborate models, or is it that we will be ever more disassociated from our own anecdotal experiences? (Where 'anecdotal' is something to apologise about).

apienx 4y ago

Being data-driven for the sake is being data-driven is indeed becoming an issue. The resources spent measuring and analysing data are overwhelmingly larger than they should in most cases. Cohorts of "data scientists" and "managers" dive head on into data without much (if any!) first-principles thinking. People tend to replicate metrics without much thought into their relevance to the specific situation. Thinking properly is a very hard skill to acquire (the hardest?), and most do everything they can to avoid it. "What you measure affects what you do. If you don't measure the right thing, you don't do the right thing." -- Joseph Stiglitz

ThomPete 4y ago

David Deutsch (Father of Quantum Computation and one of the most brilliant human beings alive) have a really great way of thinking these kinds of discussions. He calls it good explanations. A good explanation is something that is hard to vary while still solving the problem it purports to solve. He is against most use of Bayesianism when used for predictions. Great presentation here https://www.youtube.com/watch?v=EVwjofV5TgU https://www.youtube.com/watch?v=EVwjofV5TgU

tgtweak 4y ago

I've seen a lot of good arguments put to rest with a good test. The key is collecting and looking at the data correctly. Data without a keen understanding of why you need it and what you're looking to solve with it is not much use.

Alex3917 4y ago

> Be good-argument-driven, not data-driven FWIW the proper term is "data-informed."

yarosh 4y ago

1. If there are no good arguments in the collective - there's no retrospective and it's primarily a management and psychological issue. No one is able to fully self-reflect and it breaks the existing delegation / escalation chains, respectively. 2. If there are no viable data sources, when it can be proven that there's a correlation with an actual business processes, - it's a management problem. People Can't establish viable metrics, once again, mostly due to 1. This is something any company of any size and any budget can struggle with due to lack of XP and the usual collective XP-accumulation / knowledge sharing deficiency. You can't self-reflect onto something you haven't learned about, yet. And due to 1 this is a closed loop because lack of XP can't be escalated accordingly, most of the time it's also a Workplace Deviance factor. 3. Practically, it ends up in a bouquet of Workplace Deviance because no one in the end will be willing to take the blame and actual responsibility to fix anything. Any Problem vs Solution type of culture will worsen things a lot i.e. "All the blame and no Compassion". Companies are usually forced to adopt some Teal stuff in the end, maybe for really no other good reason, but just to keep on growing. The idea of hiring HR that can "work by the booK" and actually build up a personal profile of how anyone could fit into all this mess is impossible by definition - due to Employee Silence and broken retro no one will be willing to expose all the shit that is happening, in the first place... So, most of the time I see Kitchen Sink companies with volatile outcomes where there really no one who could even be able to listen to any arguments, in the first place. Google's internal ML-driven productivity metrics became a meme already for all the reasons described above. You can't reason with Toxic and Inadequate people. Also Asana claim that Social Loafing is a myth and everything else is a retro deficiency really wrong - retro can prevent and display certain glorious occasions, but it's not a root cause of any psychological effect by definition.

stuckinhell 4y ago

Be politically driven (company politics) driven. Good arguments should take in account people's ambitions, and political aspirations especially at big fortune 500 companies. Startups can be more honest.

SpicyLemonZest 4y ago

Great article, but I think it somewhat misunderstands the impetus for the concept. "Data has its place" sounds obvious precisely because "data-driven" has been such a successful concept. The alternative perspective, which used to be very common in our industry and still pops up from time to time, is that metrics are something you write for debugging and business decisions are made by gut feeling or abstract philosophical analysis. (Most software companies had to make decisions this way in the pre-cloud era, because it wasn't usually feasible to collect usage metrics.)

cptcobalt 4y ago

I couldn't agree with this more. I feel like the author took some of the arguments straight from my brain—I'm exhausted by pseudoscientific "data-driven" arguments. From my experience, most of these try to distill an incredibly complex problem space down to a one-dimensional black and white decision. But the real world doesn't work like that–it's full of grey area, and things we can't effectively measure. If you're trying to slice and dice data down to a happy one-dimensional decision point, you're often missing or ignoring important detail. At work, I'm far more happy with postmortems with general, open "good/bad" lists of after the fact feedback, that we use to consider how we prioritize and design what comes next.

ifsothen 4y ago

Yes, data is useless without a qualitative explanation. There are simply too many possible confounding factors that you cannot eliminate without understanding what they may be.

ltbarcly3 4y ago

Being argument driven gives control to the organization's 'lawyers'. People can be very persuasive independent of the reality of the situation.

N1H1L 4y ago

Be data-driven, and question the provenance of your data all the time. Otherwise you will end up like economics, a field with prettier models and more mathematics than almost every engineering field, and yet gets every major prediction wrong.

tanvach 4y ago

I’ll probably be buried in all these comments, but my position is that data is only as good as how it is collected. Sloppy data collection gives rise to sloppy conclusion through unknown biases. The key is to understand the ‘data generation process’ so you can identify biases. My experience suggests that doing so side-step some common pitfalls. I recommend reach out for ‘The Book Of Why’ by Judea Pearl. He includes many real life examples that’s surprisingly applicable to modern data science.

UIUC_06 4y ago

Good article. When your only tool is a hammer, every problem looks like a thumb. While we're at it: I've actually been in scrums where the "burndown rate" was analyzed as if it was actually A Thing. It is not A Thing.

colo_innerself 4y ago

A whole book was written on this very topic: "The Tyranny of Metrics" by Jerry Z. Muller https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metrics https://press.princeton.edu/books/hardcover/9780691174952/th...

romankolpak 4y ago

I have experienced this first hand, so this article resonates a lot with me. I worked with a manager who prioritized work which was easily measurable, so he could report the good numbers to leadership and get career points out of this. Unfortunately the project we took on was a demanding and technically challenging problem, and in almost a year of work of a team of engineers we made barely any real progress or made any actual difference, but the numbers were great and people were satisfied during presentations. I ended up feeling completely disconnected from my job and losing all motivation to work there.

taeric 4y ago

I don't know. There are a ton of great arguments that will lead to dead ends and stalled projects. :(

dkbrk 4y ago

I'm surprised there's no mention of Goodhart's Law [0]. Even if the metric is "well understood and free from human/social factors", once you start using it as a target that will no longer be the case. [0]: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

borski 4y ago

In Range, David Epstein talks about about NASA and some of their disasters, like the explosion of Challenger. NASA is the entirely encased in specialized knowledge, and has a completely data-driven mindset, with no room for logic. If you can't prove it with data, they wouldn't even consider it. He explains that, “Reason without numbers was not accepted. In the face of an unfamiliar challenge, NASA managers failed to drop their familiar tools... The Challenger managers made mistakes of conformity. They stuck to the usual tools in the face of an unusual challenge.” Even though the mistake that led to the Challenger disaster could have been caught, it was the uniformity of thinking that lead to an organizational blind spot, and that uniformity was to be too focused on data-driven arguments. There is a famous call prior to the disaster on which engineers had raised the concerns but it was based on intuition and a few cherry picked samples, not a full set of data, and this was the night before the launch. Because of the lack of data, they went ahead with it and we all know the tragedy that ensued. Moreover, other engineers who agreed that there was an issue didn't speak up, because they too lacked the data, and knew that management wouldn't care.

blueyes 4y ago

Key idea is the "data maturity" of the topic under discussion. Where there is data, you should use it and be smart about it. For a lot of big decisions, especially in companies doing something new, there is no good data at first. You have to reason about it based on experience and analogy. Then, once you commit to a path, you can start gathering data to see if your hypothesis was correct. The further you go, the more you can rely on data, assuming you know how to think about it. Discussions about being data-driven that don't take into account the "data maturity" of the situation are nonsensical. Being "data driven" when you're considering something radically new is either delusional or a cop out. Ignoring data when it could correct your biases is either lazy or wrong or both. And finally, lots of people who claim to be "data driven" are not smart about data. To paraphrase Wilde, "data is rarely pure and never simple." It doesn't just reveal truths you can treat as dogma. It's ambiguous and takes a lot of work to interpret. A lot of "data driven" teams aren't doing that work.