8 ms·
Social sciences suffer from severe publication bias
- iamwithnail 12y agoIt's an interesting point, but it's a tough one. Real example: I'm a social scientist, and I have a hypothesis about a better method of analysing discourse (borne from theoretical musings, mostly). Once I spend the next 6 months testing that empirically, and it turns out to be nonsense or completely unfounded, I'm not sure how much value there is in putting it out there - a lot of social science hypotheses are (imo) quite ephemeral and move on quite quickly. Part of that's a spillover from the fact that it's difficult to scale qualitative data gathering and analysis as an (often individual) researcher, and a frequent assumption that only quantitative data 'counts', but yeah. Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work.
- onion2k 12y agoI'm not sure how much value there is in putting it out there There's a huge amount of value in sharing 'failed' experimental data and analysis (so long as the experimental methodology was sound). People can learn new things from meta-analysis of failures and correlating disproved theories against proven ones. In fact, there are journals dedicated to it; http://www.jnrbm.com/ http://www.jnrbm.com/ and http://www.arjournals.com/ojs/ http://www.arjournals.com/ojs/ are examples.
- sp332 12y agoExactly! A negative result is not a failed experiment, it's a successful experiment. Think of all the 0's and 1's in your computer files - now imagine if someone just deleted all the 0's because they were boring. Negative results have just as much information as positive results, if you did the experiment right.
- JoeAltmaier 12y agoNot quite. As information they are much, much weaker. There are an almost unlimited number of negative results, and only a few positive ones. Imagine you are tuning a radio with a dial. Nobody wants to hear 'static at 100.3MHz! Static at 100.31MHz! Static at 100.32MHz' They'd rather hear 'Signal at 103.5MHz!'
- sp332 12y agoHow would you know there is no signal at 100.3MHz if you didn't look? Edit: A standard FM radio filter is +/- 90kHz around the tuned-in frequency, so if you choose 100.3 on your dial, you're really getting everything from 100.21 to 100.39 all at once. And 100.5 is really 100.41 to 100.59, so we're really only skipping 20% of the frequencies.
- lutusp 12y ago> Not quite. As information they [negative results] are much, much weaker. No, that is false, and further, it distorts entire fields. In psychology, for example, phony mental illnesses like Asperger's become established as real things only because of all the positive results, not accompanied by an equal number of negative ones that were never published. And in drug therapies, many drugs are validated and sold only because there are only positive published results. In one correction to this trend, the FDA funded a study in which all studies of a given class of drugs -- published and unpublished -- were analyzed in a meta-analysis. The result was that the statistical results fell below clinical significance for the majority of subjects: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050045 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... "Drug–placebo differences increased as a function of initial severity, rising from virtually no difference at moderate levels of initial depression to a relatively small difference for patients with very severe depression, reaching conventional criteria for clinical significance only for patients at the upper end of the very severely depressed category."
- JoeAltmaier 12y agoOf course negative results are valuable. But as signal-to-noise there have to be many, many more of them to help define what is there by showing what isn't. They have individually less weight than a positive value, is the argument.
- lutusp 12y ago> Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work. But repeating failed experiments has the same scientific value as repeating successful ones -- it solidifies the result, increase our certainly about it. In real science, null results, and their replication, are extremely important. My favorite example is the Michelson-Morley experiment that expected to find differences in the speed of light due to the motion of the earth through the hypothetical ether. The failure of this experiment, its publication, and efforts to replicate the result, also dutifully published, finally created a revolution in physics when Einstein explained why there was a null result, and coincidentally gave birth to the modern world.
- iamwithnail 12y ago'Real science' - thanks for that. ;-) I think the issue I'm trying to get at - I suspect there is less value in null proving a lot of social science hypotheses - the impacts are smaller than physical constants of the universe (or finding say, that the Higgs Boson didn't exist) - if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. I guess though, the main issue is: I'm sure as hell not going to get published saying it, which ties back nicely to the point made in Nature, really.
- lutusp 12y ago> I suspect there is less value in null proving a lot of social science hypotheses Yes, but that applies equally to proving positive social science results. My point wasn't to compare social science to other sciences, it was only to point out the value of replicating null results in general -- that it stands equal to replicating positive results. > if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. Both findings have equal weight, therefore both findings deserve equal attention from replicators. By contrast, ignoring null results is a disaster for whatever science they occur in -- it may mean that other people might spend a lot of time spinning their wheels studying things that have already been found wanting.
- dspillett 12y ago> I'm not sure how much value there is in putting it out there I'm sure there is great value in putting it out there if people find it later. The problem is two fold: firstly where do you put it (journals with limited space and limited time for reviewing content aren't going to be all that interested, and people aren't going to trust some random page on the Internet), and secondly will people even look for it before starting their work? This affects many scientific area and not just this one - just think how many instances there have been of medical experiments going ahead based on the results of a study or two despite the fact that a bunch of other studies contradicted the findings? Those studies either were not where they could be easily found, were not searched for, or were not published at all at the time. This is particularly problematic when there is a conflict of interests due to studies being commissioned by those selling the products/processes/ideas they hope the studies will support: there is a commercial drive to publish every success and down play every less positive result.
- harvestmoon 12y agoMaybe this type of issue would be ideal for a highly curated, online database. At the moment, for instance, I've done a lot of research into biomedical topics, and get pretty much all my info through online pdfs linked to from PubMed. (Though, of course, most of those have been published in a reputable journal.) Still, if the content was both checked for being reasonable and also classified well, then this type of online database could possibly provide a lot of value to researchers.
- p4bl0 12y ago> I'm not sure how much value there is in putting it out there. A lot. Failed experiments can be the basis for an improved one attempting to fix what made it go wrong the first time, even if "only" for a master thesis study. It can also be a real asset for anyone writing a paper on a similar subject with working results since they can cite the failed experiment to quickly justify why they didn't test some variations of their hypothesis etc. In a world where paper are often limited in number of pages, being available to list failed attempts with citations is awesome compared to having to write a small paragraph with approximative and most often bad explanations for each failures. A first step might be to put such (non-)results in short papers and upload them to an open access repository in the kind of arXiv or HAL for instance.
- harvestmoon 12y agoAssuming that your idea is reasonable, then there could be a lot of value. What if I am a researcher who has a similar idea and is saved 6 months testing because of your work? Or what if I read it and see that some of it is valuable and look into a related area. Of course, if there is a tremendous amount of research published on, say, how to make a perpetual motion device, I can see how that could waste people's time.
- ExpiredLink 12y agoThere is a deeper problem with research in Social sciences. It lacks theoretical underpinning. Studies and experiments are frequently based on 'pragmatic', ad-hock assumptions that are easy to confirm but irrelevant.
- stewbrew 12y agoI'd say this is a biased view. In German speaking countries, social sciences rather suffers from theory flim-flam. Maybe in the US, you didn't have your Luhmann-moment yet.
- 1337biz 12y agoCan you elaborate on that?
- lutusp 12y agoThe prior poster lamented the absence of validated theory in the social sciences. Your point is that many of the theories that do exist have no validity. I think those views dovetail.
- ExpiredLink 12y agoExactly.
- stewbrew 12y agoBased on your original comment I had the impression our views rather diverge but maybe I got you wrong. Anyway, HN isn't really the place to discuss such matters.
- iamwithnail 12y agoI'd agree with Stewbrew. SocSci is PACKED with people trying to jam new theory into already crammed areas, because it's the only way they'll get on; although I agree that the studies and experiments are based on sometimes fleeting assumptions or hypotheses, it isn't because of a lack of theory.
- PaulHoule 12y agoSo far as this is concerned peer review may be part of the problem, not the solution.
- short_circut 12y agoHonestly I feel like this is an even bigger problem in the natural sciences. In my experience journals want strong and interesting positive results. They rarely want negative results unless it is debunking a previous result that seemed interesting and positive. I have worked on projects before, that while interesting and on a solid theoretical base as far as approximations go, were ultimately incorrect for what ever reason. I should be able to publish and report on this so no one else wastes their time on it or can find a glaring flaw in it and build upon it. While the initial idea may be a good one the implementation of that idea is not always a good one. It can be useful to other scientists to know these things.
- CWuestefeld 12y agoThis suggests to me that there might be a follow-on problem with fabricated results. Given the "publish or perish" thing in academia, and devaluation of negative results, it seems there's a strong incentive to obtain positive results. And this is likely to increase the incidence of researchers being pressured into tweaking data to obtain affirmative results.
- evanwolf 12y agoAnyone want to join me in starting a Null Results Journal? Honor hard work with meh outcomes.
- untilHellbanned 12y agoOnarbor, https://onarbor.com https://onarbor.com, has got you covered.
- lutusp 12y ago> Anyone want to join me in starting a Null Results Journal? Nice thought, but there's already the JIR, the Journal of Irreproducibe Results: http://www.jir.com/ http://www.jir.com/
- capnrefsmmat 12y agoPLOS ONE explicitly accepts null results as long as they're methodologically sound.
- skywhopper 12y agoI like that it says "studies show" that null results are rarely published. In fact, from my friends who are in the social scientists, that's just how the journals operate. Sure, null results are meaningless for complex hypotheses, but when testing a very narrow variable with a well-designed experiment, a null result is actually a very important result.
- lutusp 12y ago> Sure, null results are meaningless for complex hypotheses ... If that's true, if a hypothesis is vague enough to resist meaningful testing, then a positive result would be equally meaningless. But I see we are in agreement.
- gwern 12y agoGelman comments: http://andrewgelman.com/2014/08/28/publication-bias-social-sciences-unlocking-file-drawer2/ http://andrewgelman.com/2014/08/28/publication-bias-social-s...
- gwern 12y agoFulltext: https://pdf.yt/d/ZTzcsTMzDeUPKhiq https://pdf.yt/d/ZTzcsTMzDeUPKhiq / https://dl.dropboxusercontent.com/u/243666993/2014-franco.pdf https://dl.dropboxusercontent.com/u/243666993/2014-franco.pd... / http://sci-hub.org/downloads/ce4a/franco2014.pdf http://sci-hub.org/downloads/ce4a/franco2014.pdf
- plg 12y agoRigorous, well-concieved hypothesis-driven research with lots of controls, ought to be considered as valuable as studies that show positive findings. The sad fact is that they are not. There is also a chicken-and-egg problem here with changing the culture. Sure, I could decide tomorrow that I will no longer battle for space in the high-impact journals that usually display this bias — in favour of more enlightened journals like PLoS ONE ... and I wouldn't suffer, I have a job... but what about my grad students and postdocs? The consequences for them would be real, because biases also still exist in the minds of professors, hiring committees, and even their peers. 10 papers in PLoS ONE just are not considered the same as 10 papers (or even 5 papers) in more "established" journals. There is thus a lot of implicit pressure to just abandon studies that show null results and pursue different ideas until you find an effect. There is also huge pressure today (very explicit) to continuously generate "novel" ideas. If you want a competitive research grant now you have to generate a groundbreaking idea. If you want to publish in the top journals you have to generate groundbreaking ideas. Amazingly there are colleagues that have a success rate with grants and with high-impact publications so regular (EVERY SINGLE YEAR) ... that wow, how amazing it must be to be able to generate a brand new groundbreaking idea at such regular intervals!!! (sarcasm). And so there is yet another kind of bias present. (politics, the favoured few, the ivy-colorer-special-people, etc) I could go on with other injustices but the bottom line is that [NEWSFLASH!] science is not objective, but like all aspects of life, is chocked full of biases, politics, special interests, and cow-towing to the bottom line (i.e. money).
- lutusp 12y ago> I could go on with other injustices but the bottom line is that [NEWSFLASH!] science is not objective, but like all aspects of life, is chocked full of biases, politics, special interests, and cow-towing to the bottom line (i.e. money). s/science/the contemporary practice of science/ Science hasn't changed -- it means the same thing. Its practice has changed to the degree that we may be justified in calling some of it pseudoscience.
- plg 12y agoyes absolutely thanks for clarifying what I meant