32 ms·
What's wrong with social science and how to fix it (2020)
- ThrowawayTestr 4y agoIf I had a bunch of money, I'd setup a grant specifically for funding replication studies. Real science can be replicated and the only way to replicate is to do it.
- amelius 4y agoAlso, replication studies could be a great way for young scientists to get started in their career.
- FeepingCreature 4y agoI mean, the incentives do seem to blow the other way. "We have invalidated the well-paid and instrumentally useful papers of a dozen other scientists! Please hire us!"
- amelius 4y agoIt may seem that way, but there will always be a PI involved, with different incentives.
- PartiallyTyped 4y ago> Hey we found those issues in so and so papers, we identified mistakes of a dozen other scientists and we do our due diligence. Please hire us! I think it's a lot about the phrasing.
- zinclozenge 4y agoWhen I was doing my physics masters (at a pretty good school ie had a couple of nobel prize winners) there was at least a handful of students whose phd projects involved "testing" theories of gravitation and pushing them to their limits. Not quite the same as experimental replication, but along the same lines I'd say.
- kansface 4y agoGiven that they are testing theory, yes. What theory underlines psychology? How does one relate the findings?
- texaslonghorn5 4y agoYes, and this is already how many PhD students spend their first year or two
- akira2501 4y agoIf the incentive is to get the grant and not to actually produce replicable studies, then what impact would you expect this to have? If the inputs are mostly garbage, then all this might do is confirm what we already know. I wonder if you'd be better off having a two-phase grant mechanism. One smaller payment for the research, and a second larger payment if it is replicable. Actually incentivize the "market" to produce good work. As it is, this incentive doesn't actually exist.
- yucky 4y agoThere needs to be a prestigious award with a large monetary reward for scientists who definitively disprove the most previously published studies each year. Then track scientists how have their names attached to the most garbage studies and publicly drag them. Best way to clean house on an untrustworthy institution that's been politicized. Do science for the science, not to fit some shit narrative.
- themitigating 4y agoScience is not an organization, business, or any single entity and discussing it as such is ridiculous. How would you implement changes?
- madsbuch 4y agoIt is an institution though. With very strict rules regarding the scientific method.
- orhmeh09 4y agoThese rules are not as strict as one might think, especially when they’re tied up with people’s careers—consider the reproducibility crisis for instance.
- madsbuch 4y agothey ought to be, though not all people have the integrity.
- madsbuch 4y agoI am actually following a course right now. Self paid. It costs roughly USD 1000 to follow and is done by a renowned University. The lecturer just released an exam which is utterly unreadable. He is not a native english speaker. But there are several grammatical errors. The progression is bad, etc. He introduces non-existing terms (Like innovation application) and is incoherent. Obviously the teacher did his best. But i really think it begs a question to whether that is good enough. People should not be doing things they are not competent at.
- ivalm 4y agoNo, there absolutely no strict rules regarding the scientific method. At most there are conventions about the structure of papers in some journals.
- madsbuch 4y ago
- timcavel 4y ago
- davidgay 4y agoThe actual title is "What's Wrong with Social Science and How to Fix It". Not exactly a neutral title edit... Especially as this comes from "a part of DARPA's SCORE program, whose goal is to evaluate the reliability of social science research"
- class4behavior 4y agoThe original title is too long and I assume op thought the findings can be extended to all science anyway. On the other hand, removing the later part as the previous submission did makes the title sound overly presumptuous, imo.
- pkkm 4y ago> The actual title is "What's Wrong with Social Science and How to Fix It". Yeah. I couldn't fit the whole thing into the hard 80-character limit so I decided to drop the word "social" because I thought the subtitle was important to keep. I guess it's moot now that the title has been edited, I assume by a mod.
- Semaphor 4y agoNeeds (2020) 100 comments at the time: https://news.ycombinator.com/item?id=24447724 https://news.ycombinator.com/item?id=24447724
- yarg 4y agoReplication is a huge issue, but I've been wondering lately about refusal to publish. Even if all of the published papers were peer reviewed and replicated, there's a lot of science that never sees the light of day. Even non-results are important - if not glamourous, and (even worse) publishing is disincentivised if the results go against what the funders wanted or expected.
- trompetenaccoun 4y agoThe replication crisis can not be overstated though because "science" that can not be replicated isn't science at all. The very base definition of science is that it's a method that produces testable and predictable propositions about our world. If a theory or paper does not deliver that it's guesswork, a superstition or religious belief in extreme cases. But certainly not science, no matter how many people or institutions refer to it as such. Just as the Democratic People's Republic of Korea isn't actually democratic. Do not trust the science. Verify the science.
- jltsiren 4y agoScience is about the process, not the results. Somebody discovers an phenomenon, studies it, fails to take something relevant into account, and publishes a faulty result, and that's science. Somebody else tries to replicate the study, fails it, and can't figure out what went wrong, and that's science. Then somebody discovers the flaw, gets a different result, and publishes it, and that's science.
- trompetenaccoun 4y agoWhen there is no reliable review mechanism, then the process is inherently flawed. It may be science in your book but following that definition, everything anyone studies and publishes is science. Homeopathy is science by that definition. Flat earth studies too. There needs to be proper scrutiny to ensure minimal quality standards are at least followed, else the whole thing is bunk. A study that can not be replicated by anyone is not research, it's monkeys hitting keys without understanding what they're doing. Sadly monkeys with PhDs in some cases but that doesn't change anything. The distinguishing feature of what "real science" is in the eye of the believer seems to be someone holding an academic title, i.e. an argument from authority, not verifiable standards that exclude randomness. There is nothing scientific about belief.
- christkv 4y agoThe worst part is when these studies lead to policy changes.
- hirundo 4y ago"Yes, you're reading that right: studies that replicate are cited at the same rate as studies that do not ... Astonishingly, even after retraction the vast majority of citations are positive, and those positive citations continue for decades after retraction." The academic influence of social science papers is uncorrelated with their scientific quality.
- Der_Einzige 4y agoIn some communities, notably AI, there are attempts to fight this which are quite successful: e.g. https://paperswithcode.com/ https://paperswithcode.com/ and the recent phenomenon of people hosting their system demonstrations on huggingface.
- matthewdgreen 4y agoReading through this quickly I see several problems that make me wonder if the author has worked in academia. 1. Publication venues aren’t created equal. People outside of academia don’t understand that anyone can and will start a journal/conference. If I want to launch the Proceedings of the of Confabulatory Results in Computer Science Conference, all I need is a cheap hosting account and I’m ready to go. In countries like the US this is explicitly protected speech, and the big for-profit sites run by Elsevier et al. will often go ahead and add my publications to their database as long as I make enough of an effort to make things look legitimate. In any given field there are probably a handful of “top” venues that everyone in the field knows, and a vast long tail ranging from mid-tier to absolute fantasist for-profit crap. If you focus on the known conferences you might see bad results, but if you focus on the long tail you’re doing the equivalent of searching the 50th+ page of Google results for some term: that is, you’re deliberately selecting for SEO crap. And given the relatively high cost of peer-reviewing the good vs not peer-reviewing the bad, unfiltered searches will always be dominated by crap (surely this is a named “law”?) Within TFA I cannot tell if the author is filtering properly for decent venues (as any self-respecting expert would do) or if they’re just complaining about spam. Some mention is made of a DARPA project, so I’m hopeful it’s not too bad. However even small filtering errors will instantly make your results meaningless. 2. Citations aren’t intended as a goddamn endorsement (or metric). In science we cite papers liberally. We do not do this because we want people to get a cookie. We don’t do it because we’re endorsing every result. We do it because we’ve learned (or been told) about a possibly relevant result and we want the reader to know about it too. When it comes to citations, more is usually better. Just as it is much better to let many innocent people go free rather than imprison one guilty one, it is vastly better to cite a hundred mediocre or incorrect papers than to miss one important citation. Readers should not see a citation and infer correctness or importance unless the author specifically states this in the text at which point, sure, that’s an error. But most citation-counting metrics don’t actually read the citing papers, they just do text matching. Since most bulk citations are just reading lists, this also filters for citations that don’t mean much about the quality of a work. The idea of using citation counts as a metric for research quality is a bad one that administrators (and the type of researchers who solve administrative problems) came up with. It is one of those “worst ideas except for any other” solutions: probably better than throwing darts at a board and certainly more scalable than reading every paper for quality. But the idea is artificial at best, and complaints like “why are people citing incorrect results” fundamentally ask citations to be something they're not. Overall there are many legitimate complaints about academia and replicability to be had out there. But salting them with potential nonsense does nobody any favors, and just makes the process of fixing these issues much less likely to succeed.
- boxed 4y agoA few of the suggestions are about making new financial incentives. But the problem in the first place is the financial incentive structures! It's much better to let people do their jobs and not have a bunch of incentives. We know this is true for programmers, why do people expect it's not true about everyone else?
- robertlagrant 4y agoGenerally people do work for money.
- boxed 4y agoYes. But not per line of code. Or per bug fixed.
- poszlem 4y agoBecause if you don't add money as incentive the only other strong incentive is for people who join the social sciences because they want to push some kind of policy that they feel strongly about (In this case, their personal beliefs and values serve as their incentive). There is even a joke among psychologists that people who join psychology departments are mostly those who themselves need psychotherapy (and they choose to study psychology instead).
- pclmulqdq 4y agoThere are lots of incentives to be a programmer today, and incentives are known to be involved in engineering decision making (hence the common criticism of resume-driven development).
- boxed 4y agoThe incentive is "have a job". Idiotic incentives are "paid per number of lines written", "paid by bugs solved", etc. That's what I'm talking about.
- 10g1k 4y ago
- marcosdumay 4y ago"Science" is the name of how you work with something, not of a subject.
- 10g1k 4y agoAnd there is no science in social science.
- yucky 4y agoYes, which I believe is why people rightly dismiss social science, as not being scientific at all.
- orwin 4y agoWhat is the science process that social scientists do not follow? Hypothesis refutability? Replication? (probably their weakest point btw) I mean, in this thread i've read about successes in sociology and economics, probably the most math-based social sciences, but i'd say linguistics are probably the second most successfull science in the last 40 years, and that's considered a social science. History have also made a lot of new discoveries recently (by criticizing older works and getting past them). Archeology have slow successes, but still, we understand Neandertal way, way better than we did ten years ago.
- einpoklum 4y ago> the most math-based social sciences Please do not confuse "with papers using math which is cavalierly assumed to model reality well" with "math-based".
- dash2 4y agoSigh. This over-the-top reaction is very common among people who don’t know better. I’ll just link my answer from the last time (here, about economics, but it works fine for soc sci in general): https://news.ycombinator.com/item?id=31411593 https://news.ycombinator.com/item?id=31411593
- patientplatypus 4y agoThis has always been the part of academia I have not understood. Academics are supposed to "know" more than the public. They are paid to have some understanding of reality in a way that the lay public does not so they can have tenure - that is they can't be fired (or it is very hard) for researching controversial subjects with payoffs that are hard to define by someone not in the field. Everyone else in academia is working towards this end result or drops out somewhere along the path and works in industry. The exception being those specialist degrees that are quasi-academic such as law and medicine, but which have more applied (and therefore measurable) results. And yet, we don't hold these people to the standards they set for themselves as a cohort? I wouldn't expect every paper to be replicable, or every researcher to always be right. But I see things like the https://en.wikipedia.org/wiki/EmDrive https://en.wikipedia.org/wiki/EmDrive and wonder how many billions of dollar were put into this thing. It's essentially a state backed way of defrauding the public out of tax dollars to pay people with doctorates that made friends with other people with doctorates. What if the US spent the money that was used on the EmDrive on housing or social services for instance? Or building libraries or bridges? All it would take would be for a few academic journals to demand that the statistical replication of papers be required, but it would mean that the editors would have to be guided by principles other than maximizing revenue.
- poszlem 4y agoWe absolutely need to change the way we do social sciences. My favourite pet peeve in social sciences is idea laundering. "It’s analogous to money laundering. Here’s how it works: First, various academics have strong moral impulses about something. For example, they perceive negative attitudes about obesity in society, and they want to stop people from making the obese feel bad about their condition. In other words, they convince themselves that the clinical concept of obesity (a medical term) is merely a story we tell ourselves about fat (a descriptive term); it’s not true or false—in this particular case, it’s a story that exists within a social power dynamic that unjustly ascribes authority to medical knowledge. Second, academics who share these sentiments start a peer-reviewed periodical such as Fat Studies—an actual academic journal. They organize Fat Studies like every other academic journal, with a board of directors, a codified submission process, special editions with guest editors, a pool of credentialed “experts” to vet submissions, and so on. The journal’s founders, allies and collaborators then publish articles in Fat Studies and “grow” their journal. Soon, other academics with similar beliefs submit papers, which are accepted or rejected. Ideas and moral impulses go in, knowledge comes out. Voilà! Eventually, after activist scholars petition university libraries to carry the journal, making it financially viable for a large publisher like Taylor & Francis, Fat Studies becomes established. Before long, there’s an extensive canon of academic work—ideas, prejudice, opinion and moral impulses—that has been laundered into “knowledge.” (source: https://www.wsj.com/articles/idea-laundering-in-academia-11574634492 https://www.wsj.com/articles/idea-laundering-in-academia-115...) I was one of the extreme "trust the science" people, until I joined a startup that worked with academia. The amount of pettiness, vindictiveness, cutthroat power games I have seen surpassed even the most hardcore of startups.
- geraldwhen 4y agoI struggle to see what part of social science is morales or quasi religious beliefs mascarading as science. There is no cure.
- poszlem 4y agoAt the very least we need to start demanding two things: falsifiability and reproducibility. We should stop calling "science" things that are unfalsifiable.
- analog31 4y ago>>>> Increase sample sizes and lower the significance threshold to .005. I'm a physical scientist in industry, so I don't publish, or use p-values. But if the familiar "square root of n" rule holds, than a 10x improvement in any kind of "noise" measure requires a 100x increase in sample size. This is probably impossible for many kinds of experiments. Even in my work, desiring such an improvement usually requires me to scrap an experiment and come up with a better one, rather than slogging away to collect more data. Worse, we may suffer from the fact that for some kinds of studies, sample size is limited by the number of potential subjects per researcher. You can only ask undergraduates to do so much. More researchers in the field means weaker statistics. A possible minor improvement is to eliminate the requirement of publishing a masters thesis project. My friends who were psych grad students all had to do a master's en route to the PhD, and it had to be published to fulfill their requirement. This might have seemed like a quality check, but instead created an incentive to load the literature with low quality research. And many of the students were not looking forward to research careers anyway. Their term for finishing the degree was "published and outa here."
- Diapason 4y agohttps://youtu.be/kVk9a5Jcd1k https://youtu.be/kVk9a5Jcd1k
- addingadimensio 4y agolol is this the first right wing thread on hacker news? Are the plebs finally waking up?
- Haga 4y ago
- mike_hearn 4y agoIt's a great article which unfortunately makes it sound like the problems are restricted to social science. That isn't the case, similar problems crop up in many other fields (or perhaps you could argue the scope of social science is wider than appreciated). A good example of a field with the same problems is epidemiology. I wrote a brief article with many examples a couple of years ago [1]. Unfalsifiable claims, objectively false claims, non-replicable work, citations that were invalid in various ways (e.g. citing a paper in support of a claim which didn't actually support the claim), and most surprisingly, a lot of circular logic. This field seems to routinely lose track of what comes from assumptions vs simulations vs reality. Papers would make claims that sounded like factual statements about observed data, but when the citation was checked the number would turn out to be a simulation output or even an arbitrary assumption used as simulation input. Validation had gone AWOL long ago with the substitutes being unscientific, e.g. claiming a model was validated because it roughly matched the output of other models. You could argue that epidemiology is actually a social science rather than a hard science. Maybe the distinction isn't interesting though. [1] https://plan99.net/~mike/epidemiology.pdf https://plan99.net/~mike/epidemiology.pdf