11 ms·

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going t

• throwawaysea 5y ago
  There was a past discussion about peer review on HN: https://news.ycombinator.com/item?id=20607259 https://news.ycombinator.com/item?id=20607259 Another related discussion was about the grievance studies scandal, which also touches on peer review and academic rigor in journals: https://news.ycombinator.com/item?id=18127811 https://news.ycombinator.com/item?id=18127811
• javert 5y ago
  The incentive system that academic scientists live under explains why they don't push for studying metascience. If that's going to happen, it has to come from outside the government-science complex. Good luck with that...
• ryanmarsh 5y ago
  Similarly, I heard someone the other day assert that no one had done an double blind placebo controlled trial of the effects of FDA regulation.
  • hannob 5y ago
    That is probably true, but do you have any proposal how such a trial should be done? RCTs are good when they can be done and I'm all for doing more of them and too often there's no good excuse for not doing them. But at some level things just get impractical.
• logicchains 5y ago
  >When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science? In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.
  • hpoe 5y ago
    I think this is an important point, ultimately good science will produce, verifiable, testable, actionable results. Until you have that no matter how much math you use, how many lab coats you've got, no matter how many journals you publish in you're just sitting there playing with strings.
    • jrochkind1 5y ago
      Telling when this has happened can be less trivial than you would assume though. People (scientists even) were sure that phrenology produced verifiable, testable, actionable results for a generation or two. In the long run it usually comes out, but the run can be longer than you think, and you may not be where you think in it with regard to any particular current theory. I wonder what things we know all "know" are proven by science will be dismissed by later generations. (I personally guess a lot of genetics-related stuff will be). (Note that something doesn't need to be verifiable, reliable, or true to be "actionable". You can act on anything...)
  • jhap 5y ago
    > In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money. I think this is a common critique, but I also think it is missing the point. What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social-sciences-are-useless-so-why-do-we-study-them-heres-a-good-reason/ https://statmodeling.stat.columbia.edu/2021/03/12/the-social...). Finance is also a bit tautological in this regard. It seems that often prediction models are impossible to disprove (e.g., our arbitrage method doesn't work anymore, the market updated). Yes good for putting skin in the game, but doesn't seem like it does much to advance our long-term understanding of humans.
    • logicchains 5y ago
      >What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social https://statmodeling.stat.columbia.edu/2021/03/12/the-social...). Some things may well be complex enough that it's simply impossible, with the amount of resources available to the average university, to conduct a thorough enough study on a representative enough sample that accounts for enough confounding factors to make a statistically sound prediction that generalises. If this were the case for a significant proportion of the subjects of study of a particular field, then it might well be better to "give up" and admit we don't and cannot know, otherwise we're essentially creating a factory for bad science (as the available resources relative to the scope of the problem aren't sufficient to create good science, and there's no negative feedback to stop the bad science).
• munificent 5y ago
  > it's insane that metascience isn't studied more. It is, probably more than is apparent. See: https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Philosophy_of_science https://en.wikipedia.org/wiki/Sociology_of_scientific_knowledge https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle... There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you are trying to understand. Also, as any sociologist will be happy to tell you, incentive structures and other human group behavior gets in the way. It's probably hard to get funding for a study that shows that all the other departments at your university aren't quite the flawless seekers of truth they appear to be.
  • chromaton 5y ago
    >what tools do you use to scientifically analyze the scientific process? Iterating on existing systems to see if you can get results to converge and also testing new systems to see if they also result in known good values.
    • licebmi__at__ 5y ago
      That seems broad enough to describe the human history isn’t it? So, “keep doing what you do”?
      • chromaton 5y ago
        Not necessarily. You could just fall back on authority, superstition, or navel-gazing.
        • dpe82 5y ago
          That would probably fall under "testing [new/other] systems to see if they also result in known good values."
    • munificent 5y ago
• bluetomcat 5y ago
  "People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means that the study wants to build "universal" models of how each single person in the whole world acts, which is utterly dismissive of the specific local environment around people. Sociology in particular should always be approached highly critically, because applying those theories and reasoning in its terms often means mass control over people's free will.
  • hpoe 5y ago
    Honestly all sociology I have seen or been exposed to, including in college, seems to be more interested in acting as a platform to push specific ideas, rather than an attempt to find truth. Beyond that those involved in sociology seem to believe that a study is the same thing as an experiment and like to believe that constitutes proof. Ultimately we can't really run AB experiments on society at large because we are living in; however humanity has at its disposal all of history as a case study. My point is if you really want to understand how societies interact and form, and react, and live ask a historian, not a sociologist. I also would apply most of these comments to economics except there seems to be more diversity of viewpoints, and studies are used less than math to try and provide a veneer of respectability. EDIT: If someone feels that history is inferior to sociology for understanding how societies act and behave please tell me why. I want to understand where I am wrong. But I see a lot of our arguments that we are having in society nowadays the same as one's had a thousand years ago, the discussions over Social Media are basically the exact same ones people had over the printing press in Europe, I recently read "The Republic" and there were the exact same arguments I see repeated here. So if you feel contrary please tell me why, I admit I could be wrong, but want to understand where my reasoning is flawed.
    • notsureaboutpg 5y ago
      I felt this way as well. But you might benefit from reading more old school sociology books. C. Wright Mills The Sociological Imagination is great (should have been taught in college to you). Thorstein Veblen's Theory of the Leisure Class is good as well. These really seemed to me like attempts to approach truth, and perhaps that's because of the time they were written in vs the time we live in now.
• libraryofbabel 5y ago
  There is a ton of current and past scholarship on the sociology, history, and philosophy of science, and peer-review and publication is actually a pretty hot topic in those fields. Although yes, perhaps it would be nice if that work was better funded, or if practicing scientists paid more attention to it rather than just repeating old myths about how science works. For example, here’s a scholarly article on the exact question you mention - how and where peer review came to be seen as a guarantor of scientific quality: https://www.journals.uchicago.edu/doi/abs/10.1086/700070 https://www.journals.uchicago.edu/doi/abs/10.1086/700070 (tldr: it wasn’t the 17th century Royal Society; it’s much more recent.)
  • aeturnum 5y ago
    Another classic on this subject is Shapin's "Pump and Circumstance" - which is also freely available: https://scholar.harvard.edu/files/shapin/files/shapin-pump_circum.pdf https://scholar.harvard.edu/files/shapin/files/shapin-pump_c...
• rvba 5y ago
  Current "science" is often not interested in replication or even testing if experiments can be replicated. The culture is tk be quoted and "science but boring" is not quoted. On top of that there is publish or perish. Also dont want to point fingrers but some scientists come from places where cheating is the norm.
  • deleted 5y ago
    [deleted]
• tgbugs 5y ago
  It isn't so much that it is not applied as that the pressure to generate data inside academic science makes it extremely difficult for grad students and postdocs to allocate the time to doing science on their experimental processes. They already only barely have time to do experiments on the system they are actually trying to study! Engineering organizations inside major corporations usually actively engage in process improvement because they are resourced to do so.
• Bayart 5y ago
  Epistemology is an interesting field, but it's just *not part of the curriculum for engineering and experimental sciences types. I only had a good introduction to it when I took it as an optional course in a humanities college.
  • WalterGR 5y ago
    *just not part
    • Bayart 5y ago
      Indeed !
• pacbard 5y ago
  I saw a chart once that made the point that most research studies can be classified along two axes: rigorousness of methods and popularity of results. Most published papers have either high rigor/low popularity and low rigor/high popularity. The trade-off is in the fact that highly rigorous studies only allow for narrow, unexciting results, while popular studies with flashy results will have to compromise on their rigorousness. This is not really a rule, but it is an interesting way to see research and the editorial/peer-review process. The example discussed in OP seems to fall in the category of low rigor/high popularity. I am not 100% on my history of psych research, but it seems to me that the stereotype threat was all the rage in the late 90s following the publication of Steele and Aronson (1995). OP study seems to follow a similar experimental setup as S&A with a new group of people (Asian-American women). As far as meta-science is concerned, I think that it remains mostly a part of philosophy (as in epistemology) and the focus of a few (senior?) scholars in each field. There is really no space to publish meta-scientific papers that "shake up" the field and call out established researchers, as editors that publish those pieces could come under similar criticism for their work. I think that it is not an accident that the discussion of the replication crisis in psychology started from blog posts and other non-academic avenues and then found its way to more "established" publications in the field (again, if I remember the context of those conversations). I really wish that the review process was open. It would be interesting to see the reviewers comments to this specific paper and how the editor decided to pick up and engage with them. All those conversations are usually locked up in some editorial management system and are seldom made public. I don't know if we can really have open science without having open peer reviews.
  • derbOac 5y ago
    I agree with you mostly, but it's worth noting that modern meta-science had its blossoming in psychology in the 1960s, with the development of meta-analysis (with educational psychology and clinical psychology). Technically the origins are much earlier, in the 30s(?) in statistics, but as a field I think it took off around that time, and spread. Similarly, the replication crisis was being discussed in a lot of areas, especially in psychology, throughout this time, but was largely ignored until after the Bem ESP study. Registered replications aren't new, nor is concern about meta-science; it's just had renewed focus in recent years for various reasons. It's not all that surprising to me that meta-science is associated with psychology. After all, not only is psychology often sort of fuzzy (by necessity of its subject matter), but it's the science of human behavior, which I think can lay claim to scientist behavior as well. I think it's arguably the greatest contribution of psychology to the sciences in general.
• EGreg 5y ago
  It is, but you arent aware because metameta science isn’t covered enough
• chimpme 5y ago
  super good point. it is questioned, but the doubts are mostly ignored
• aeturnum 5y ago
  If you are interested in this I really recommend you look at writings from the fields that have descended from what was called "Laboratory Studies" and now variously calls itself "Science studies", or "science and technology studies" or "science, technology and society" (STS is a common acronym). For the stuff that's very lab-focused I'd say you could start with Bruno Latour and Steven Shapin, both classics of the older guard of the field.
• spullara 5y ago
  Dedicated folks that just try and reproduce important papers would be amazing and valuable. Only time I have seen it happen at scale was during the Cold Fusion days.
  • throwawayboise 5y ago
    That's because if cold fusion had been proven to work, there were fortunes to be made. Most science has very little market value.
• jxramos 5y ago
  I was recently imagining an experiment classification tagging system: "trial", "reproduced", "peer reviewed", etc. I could imagine this set of information landing on some wikipedia page and the experiment in question would gain a bunch of these tag badges as understanding of the phenomenon matures.
• loup-vaillant 5y ago
  A good start would be something like Probability Theory: the Logic of Science By E. T. Jaynes: http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia... Also, the difference between a bad method and a good method, is that the good method makes more accurate, better calibrated predictions (that is, using it makes us better gamblers).
• logicchop 5y ago
  When someone hits you with a truck-load of mathematical jargon and convoluted experimental setups, most people just end up fatigued and find themselves nodding to conclusions out of fear of looking stupid. And the problem is only amplified by the recent "Science Rocks!" attitude making the rounds in pop culture.