7 ms·

I would say that social science is an exception rather than the rule when it comes to reliance on p-values for their results. Many, or most scientific fields do

• goldenkey 9y ago
  I have to agree. The social sciences are one place where nonsense flies free and often with wings. But even in the technical sciences, many papers don't list p-values or anything remoteley quantitative. I think we need to be careful of taking the part for the whole.
• TeMPOraL 9y ago
  Judging by the replication crisis discussions over past two years, I'd add quite a bit of medicine to that list too. And I don't mean just psychology, which is social science with a medical degree - but also drug research. All of this, though, shows how hard those disciplines are. Relative to that, hard sciences are easy, because they study precisely defined things pretty closely entangled with laws of physics, and you can't easily bullshit mother nature. But as you move away from physics, complexity rapidly increases, the questions themselves become fuzzier, and the answers less clear. So IMO, the probability that academic success is crapshoot or scam increases with (Kolmogorov) complexity of the subject domain.
  • davrosthedalek 9y ago
    In the case of at least medicine, there is another degree of hardness involved: sample size. While in physics, for example, sample size is limited by cost, space or, in few cases, frequency of occurrence, in medicine, a lot of studies are limited in size because of ethics. Many studies, especially drug trials, have an risk for at least one group. It is unethical to put more people at risk then absolutely needed, so studies are normally designed so that signal-to-noise is the minimal viable.
    • ekianjo 9y ago
      not jusT ethics. Cost is a huge factor as well for clinical trials.
      • whatshisface 9y ago
        Cost is a factor for sample size in any field: but physicists don't publish if they can't accumulate a sufficient sample. As a result physics experiments where sufficient sample sizes can't be obtained are, simply, never performed - and as a result physics as a field grows in the direction of high-certianty. Physicsts absolutely could be performing experiments with poorer statistics; it'd be cheaper, more advanced things would be done sooner, and we'd be having a replication crisis. If the social sciences were willing to go a century without discovering anything exciting, they could achieve whatever level of certianty they wanted on their current budget. (At the expense of new discoveries.)
        • magicalist 9y ago
          > but physicists don't publish if they can't accumulate a sufficient sample What? I would say most published physics is speculative models based on under-sampled phenomena (and that's ignoring the whole explicitly theoretical branches...). You could argue the practitioners tend to be more realistic about the chance of their models being valid, but this comment still ends up a bit masturbatory.
        • mathattack 9y ago
          High certainty but that comes with second order issues: many ideas are untested, and the ideas that are turn into papers with 100s of authors. This stifles independent thinking.
          • whatshisface 9y ago
            Here's a modern experimental paper. Scroll down to the authors. You'll know it when you see it. https://cds.cern.ch/record/2282544/files/arXiv:1709.05260.pdf https://cds.cern.ch/record/2282544/files/arXiv:1709.05260.pd... Your statement about huge numbers of authors (as well as the great number of theoretical papers that have to wait a very long time before being confirmed or denied) is true.
  • Retric 9y ago
    It's not quite that medicine is more complex, it's that Medicine is practical and accepts not understanding the details. If drug X works then that's enough we will use it even if we don't understand exactly why it works. The downside is we end up building simplified models well Y is correlated with bad things so we should treat Y as a bad thing. Often this actually works really well, but also opens the floodgates for all sorts of issues. That's not to say a less practical approach that simply looked for the details would be more effective. But, it would look very different.
    • denzil_correa 9y ago
      > It's not quite that medicine is more complex, it's that Medicine is practical and accepts not understanding the details. If drug X works then that's enough we will use it even if we don't understand exactly why it works. I'm always baffled by this. I can understand why we should move these drugs into practice. But, if the way you conduct research stops at X works - you should have a good look at the way you conduct research. I can completely understand short term gains but ignoring long term gains is very unscientific and immature.
      • zdragnar 9y ago
        I'm afflicted by three chronic conditions which have no known causes. All of them are treatable; there are guesses as to why the treatments work, there isn't really a good way to establish a sound reason for them working. You can't just take a person apart to study them while they're alive, and you can't easily reconfigure them to account for variables either.
      • sliverstorm 9y ago
        They'd love to understand why X works, but a whole lot of people could be treated by X while they spend all that time and money figuring it out. Besides, there's plenty of Y where they were sure it was going to work, it fit all the models and theories, it did the right thing in mice, and then in a trial Y fails catastrophically.
    • clairity 9y ago
  • gboudrias 9y ago
    > And I don't mean just psychology, which is social science with a medical degree Psychologists don't have a medical degree, psychiatrists do and they could hardly be called social scientists. Hard sciences also don't have a very high replication rate, they have the same incentives which the article talks about, and the same publishing bias problem.
  • xapata 9y ago
    Part of the replication issue is that even if the relationship exists, it's quite likely a small study will find nothing.
• JorgeGT 9y ago
  I agree, I publish in the engineering field and in the field most "proofs" are validation by simple comparison against experimental data, DNS numerical simulation, analytical expressions, etc. In some areas there isn't even validation at all, it's just "we put this interesting thing in the expensive and exclusive machine, and these are the results you guys". For instance testing different fuel mixtures in a rapid compression machine, testing prototypes in a wind tunnel/flow rig/anechoic chamber, etc.
  • skummetmaelk 9y ago
    That's actually pretty good. There is a movement towards publishing minimal results, such as those experiments. This will hopefully get rid of the need to publish only successful experiments.
• pesenti 9y ago
  It would be interesting to see the correlation between how susceptible a domain is to p-hacking and the number of papers required to be successful. I know that in Mathematics it’s ok to publish a good paper every other year.
• hacknat 9y ago
  I think your view is a little biased by your work. Much of biological research still relies on statistical methods. The Higgs boson, most of patho-biology, drug research, are just a few fields/discoveries I know for sure rely heavily on statistical models.
• learnstats2 9y ago
  No, this is attractive but plain wrong. If you look at scientific research in Nature, there are very few papers that don't mention p-values. Successfully building your biological proof-of-concept is lovely and remarkable but depends on a whole bunch of research done with p-values.
  • Cyph0n 9y ago
    No, he is not wrong. There are many fields where p-values are irrelevant. For instance, in the fields of electrical engineering and computer science, you can "prove" your results by either building/fabricating a prototype or writing a piece of software.
    • avian 9y ago
      True. Often authors don't even share the said piece of software or detailed plans for the prototype.
    • Godel_unicode 9y ago
      I wonder how true that is in data science, if you're dealing with a non-deterministic algorithm. You're going to have cases for which your magic NN doesn't recognize the cat.
    • learnstats2 9y ago
      Back to this thread to note this method of "proving" results by existence is working great for chip manufacturers right now.
• bo1024 9y ago
  Basically, you are talking about "engineering" instead of "science". Not that I'm trying to start a long discussion about definitions or mutual exclusivity, but these kind of "results" are clearly about building something new and showing it works, not proposing a hypothesis about how the natural world works and testing that hypothesis.
  • nonbel 9y ago
    >"proposing a hypothesis about how the natural world works and testing that hypothesis" The studies under question here don't really do that. Physicists often test their hypothesis (eg this satellite should undergo x seconds of time dilation), but in fields like biomedicine or psychology that is extremely rare. Instead they usually test a "null hypothesis" that there is no difference (eg the population mean of the treatment and control groups is the same) when their hypothesis predicts some vague but non-zero difference.
    • bo1024 9y ago
      The difference between testing a null hypothesis vs testing a new, proposed hypothesis does not affect my point about scientific method vs engineering.
      • nonbel 9y ago
        Sure, but I'd say there are three things going on here though. 1) Scientific method (what you originally described) 2) NHST (where some default hypothesis is tested) 3) Engineering (where you perform some kind of feat to demonstrate capability) The problems seem to be with people doing #2 rather than the others.
• bjourne 9y ago
  The problem exists in computer science too. For example, a researcher might publish an article about a novel algorithm, claiming to improve Java VM performance by X%. Except there is only pseudo code given, so replicating the result is nigh impossible. But if you succeed in implementing the algorithm, more often than not you discover that it only improves performance in the researchers specific setup and is net negative for normal workloads.
• collyw 9y ago
  As someone who has worked in biology labs doing databases for them, it's about the only statistical test I could name name as it is talked about a lot.