23 ms·
You Want to See My Data? I Thought We Were Friends
- ajuc 6y agoIs there something like negative-citation-index? Where you spread refuted papers out through citations to other scientists and newspapers. It could be included as a factor when hiring scientists. And of course the person who refuted a false paper should receive the citations of that false paper. It's only fair.
- lecarore 6y agoThis sounds like a good addition to the incentives system.
- abdullahkhalids 6y agoThe number of papers which are refuted, in the sense that a serious error is identified that invalidates the main conclusions of the paper, are a vanishingly small percentage of papers published. This doesn't mean that a vanishingly small percentage of papers are wrong, only that it is very hard to identify errors because papers usually don't contain enough information to fully reconstruct the results. There are a lot of assumptions of good will in the system.
- ajuc 6y agoThen accept papers that refute these results by providing arbitrary data where it's missing.
- crankishness 6y agoExpanding on this, suppose there is an anti-journal, tentatively titled 'Journal of Bad Science', which features thoroughly refuted, bad-faith papers. Not just rejected papers, as the reasons can be as innocuous as a few spelling mistakes, but clearly and unambiguously bad research. This would form the basis of the Crank Index of a paper, which can be simplified into a stoplight system: Good research with good sources is GREEN. Getting featured in the Journal of BS earns a paper the esteemed distinction of a blaring scarlet RED, citing a RED paper will mark a paper ORANGE, citing ORANGE research leaves you YELLOW... Throwing together lots of ORANGE and YELLOW citations will nudge your paper up the spectrum towards RED. This would incentivize researchers to not only care about the quantity of the citations they share with each other, but to be extremely vigilant of the quality of those citations as well.
- fsflover 6y agoLet’s say I cited a “red” paper and explained how and why they were wrong. Does my paper become “orange”? I hope not, but that would require a lot of rigorous manual verification by the journal editors...
- ajuc 6y agoIdeally papers would start citing using a new format that makes explicit the dependencies between papers. For example: refutes: ... expands on: ... depends on: ... alternative approach to: ... etc.
- jpeloquin 6y agoIt would be great if journals published their articles as structured data. Then readers could compute a Crank Index, or do other automated citation analysis and filtering (e.g., flag excessive self-citations), using independently developed software. We shouldn't need to wait for journals to live up to their responsibility to innovate and improve their quality control. As for a list of unambiguously bad papers, we do have Retraction Watch: https://retractionwatch.com/ https://retractionwatch.com/. It's mainly a retraction tracker, but there is also associated community effort to proactively identify research misconduct.
- Sebb767 6y agoI generally agree, but their solution (fund boring research, publish only in journals with high standard) is in direct contrast to what they stated earlier. It's basically saying yes, we have bad incentives, but we could ignore them. That's not going to happen; shiny new research will attract people and funding. Especially more than "boring" what we found before was indeed a finding-research. Now, I don't have a good solution either, unfortunately. What might work is that we require replication work for a PhD or have a certain percentage of a journal dedicated to verification. That, combined with some meta-studies to reward people with citations for replication, might work without fully swimming against the current. It's a hard problem, really.
- searchableguy 6y agoA more radical solution: Basic income for all researchers and provide efficient digital way to see how funds are being spent like open collective for open source. Most problems arise at an individual level where people don't get paid unless they are putting research papers on the new hype thing and the other is expenditure on middlemen and bureaucracy which isn't easily visible. Eliminate the use of research grants for funding people's living. Bit pedantic but you should use they instead of he/she. People can have other pronouns.
- kebman 6y agoOh, you mean tenure? UBI is a general thing for the whole population, and not something just for scientists. Tenure is in general a good, as long as scientists keep working, and there are some controls keeping them producing at least "something". And let's face it, most of them do. This is great, because it releaves scientists of pressure, which means more of them can be creative in their research, which actually speeds up discoveries. This is also a valid approach within tech and IT btw. But it should be said that it has a very poor track record outside of these knowledge-heavy professions. On the other hand, if you really think UBI is a good solution, then I'm afraid you don't know how an economy really works.
- searchableguy 6y ago
- sradman 6y agoThis comic by Zach Weinersmith summarizes Stuart Ritchie's recent book Science Fiction. To combat the problem of low-quality science papers, one of the panels suggests: > [journals] can demand scientists share their data, and to prove that they've written down their analysis plans before they touch the data I wonder if this doesn't gloss over a deeper underlying problem: journals have traditionally assumed the copyright of the paper. Journals themselves have an incentive to obfuscate and protect the underlying data and content. Ultimately, any complex system or institution will be more susceptible to gaming when it is mature and its value proposition clearly established. Anti-gamification is hard to design into the early stages of a system when it is needed most.
- adamnemecek 6y agoYou need a better science publication platform. Like arxiv and github combined.
- sradman 6y agoArxiv, github, and a self-publishing style platform that supports reproducible digital artifacts, i.e., the published paper. IIRC, many flawed papers were the result of data errors saved in a spreadsheet.
- adamnemecek 6y agoAnd better citations. I want to be able to link to a particular sentence of a particular version of a paper. Also pull requests.
- mattkrause 6y agoCould you explain what the sentence-level citation adds? I've heard several people ask for this, but never understood why. Most citation formats let you include page numbers; you can usually work in other location information ("See Foo et al. (2020)'s Figure 3A") too.
- adamnemecek 6y agoMaybe but most people dont do it.
- dijksterhuis 6y ago> I want to be able to link to a particular sentence of a particular version of a paper. Probably best that you read through whole papers instead of looking for one sentence.
- beagle3 6y agoWould help in stuff that doesn't require physical measurements - e.g. computer science, neural network results; would not help in medicine, psychology, biology or chemistry, where reactions are reported by (often indirect) observation, and it is often the data that is fudged (or just made up) in retracted papers.
- sarellaza 6y agoI get paid over $190 per hour working from home with 2 kids at home. I never thought I'd be able to do it but my best friend earns over 15k a month doing this and she convinced me to try. The potential with this is endles... Copy Here.......www.salaryapps.com
- ptero 6y agoThere is no perfect solution, but requiring access to both the data and full methodology for experimental sciences should help. Even problem sof cherry picked data would be partially exposed eventually; and eventual exposure is still a very effective deterrent in science. My 2c.
- thecreamedcorn 6y agoI find it interesting that obviously smart people (the guy who illustrated this) are unwilling to question whether science in and of itself is a noble and positive endeavour for humanity. It's always an argument like: if the scientific process was followed, or if academia was structured correctly, or if the gov didn't sponsor bad research, etc... Most of the current world population is totally oblivious to scientific advancement, every civilization before 200 years ago probably had less than percent of the scientific knowledge we have now and what do we have to show for it. We live a bit longer and die less, that's about it. There's no other impacts science has had on the human experience that's an undebatable good, so why do people insist on this grandiose idea that if we just keep following science we'll eventually be an enlightened people. But I guess that goes against most peoples preconceptions too much so just throw a panel at the beginning and end of your comic saying science is good.
- beagle3 6y agoI don't know if the comic properly represents the book, but all of the suggestions are somewhere between ridiculous and useless. First, about identifying bad research: The "extraordinary claims require extraordinary evidence" is already practiced. It's not that "we've overturned quantum theory" articles that are causing problem - those are quickly and effectively shot down. And it is rare that the "perfectly aligns with a political interest" can be applied. The only actionable one is "see what others think about it", and it's no panacea either. Bad and fraudulent science like the recently retracted Surgisphere covid paper is abundant. I was trying to track down the origin of the "reduce salt intake" and "limit egg consumption to no more than 2 per day / 2 per week" recommendation in the past, assuming there was hard science behind them. There isn't; and indeed they're slowly being reversed everywhere - but they were prevalent for half a century, with a lot of other research taking them as axioms. The remdesivir trials have been p-hacked to death - anyone who took an interest was seeing it happen in real time - yet, the scientific community turns a blind eye. The ketogenic diet is vilified in every mainstream media and most nutritional "science" publications; the headlines are rarely inline with the the actual results, but that's what people remember (Not that nutrition science is really science) And the other recommendations about fixing it are comparable to "lets solve evils of the US 2-party system! All we have to do is make those two parties vote to take away their power". Academia and science publishing are where they are now because it benefits essentially all the incumbents (at the expense of the rest of society). The problem description (at least in the comic) is good. Any suggested action ... not so much.
- joppy 6y agoI think the suggestion that "universities could change their hiring policies" is a very good recommendation. There are a lot of problems in academia (bad research, lack of diversity, pressure to publish) that could be reduced if universities changed their hiring policy to something further away from the metric of (number of papers published since receiving PhD) / (time since receiving PhD). Of course no university hires explicitly on that metric, but many of the metrics they use are not far from that, mixing in quality of journals and norms for the field, etc. Each of the suggested actions, taken together, would seem to have a very positive improvement on the status quo. Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?"
- mabbo 6y agoZach Weinersmith's ability to tell compelling non-fiction via comics is something I truly love. He manages to take what the person is saying, convert it into a comic form of them saying it, while adding humour. And throughout the process, what is being said does not seem to be degraded at all. There's also a level of openness we all seem to have to something that comes at us as a comic rather than hard text. I don't even think we have a good word for what this practice is, but I'll go with "Art" because it takes a lot of that. His book on Immigration[0] is a large-scale version of this skill in practice and I suspect a lot of HN readers might enjoy it, regardless of if you agree with his points or not. [0]https://www.amazon.com/Open-Borders-Science-Ethics-Immigration/dp/1250316960/ https://www.amazon.com/Open-Borders-Science-Ethics-Immigrati...
- deleted 6y ago[deleted]
- abdullahkhalids 6y agoOne change that would help and easy to implement is reporting, for a researcher/paper, citations excluding self-citations. This doesn't take care of citation rings, but does move the needle towards reporting the actual value of a paper/researcher.
- James_Henry 6y agoA lot of the problems in academia, I believe, come from incompetence, a lack of questioning or at least of doubting others' competence and your own competence, and reliance on unsound ideas about scientific methodology. Andrew Gelman has some good thoughts here that I feel are related: https://statmodeling.stat.columbia.edu/2020/07/29/the-crooks-get-the-headlines-but-the-real-problem-is-bad-science-done-by-non-crooks/ https://statmodeling.stat.columbia.edu/2020/07/29/the-crooks... Some people will have the best of intentions, they will really be doing "science" out of the goodness of their hearts and out of their desire to help mankind, but they'll end up publishing trash that gets acceptance and sometimes even praise. What are the incentives that need to be instilled to fix the problems we currently have? I'd say we need to incentivize competence and humility. How? I don't know exactly, but values like these seem to be able to be instilled through cultural practices and traditions. Also, especially humility seems to be lacking from many cases of bad science. If people accepted criticism and accepted that they don't really understand all that much, I believe scientific quality would improve. You do have a lot of reasons to not be humble in academia though, as this comic lays out.
- bjornsing 6y ago> I'd say we need to incentivize competence and humility. Problem is I think that the competent are few, and when the cultural norm is that they must be humble then they stand no chance against the many incompetent. IM(H)O: Science shouldn’t be humble in the face of non-science. As long as it is it will lose. The idea of conflict free great science is a pipe dream. We need a culture that accepts (intellectual) conflict.
- James_Henry 6y agoI agree that there isn't enough competence, or that "the competent are few" (though competence isn't a yes or no thing, you can be competent in some respects and not in others). However, I think that competence really only has a chance if the incompetent are humble. There will be conflicts and these conflicts should be embraced and someone who is incompetent and not humble will fight the existence of conflict rather than the actual scientific issue that needs to be solved or understood.
- im3w1l 6y agoI kindof think science isn't fixable. Maybe we can get it a little bit better but we just have to live with the suck. At least it makes some progress despite all the flaws.
- einpoklum 6y agoA couple of years ago, Prof. Michael Stonebreaker gave a talk in ICDE (IEEE Intl. Conference on Data Engineering) 2018 in Paris on the problems of the pursuit of the "LPU", least publishable unit of work; and his impression that few people pursue deeper and more significant work because of this and other factors. If you can find a summary or a recording of that somehow, it's worthwhile to listen IMHO.
- giardini 6y agoOK, so "Science Fictions" was just released and is full of cartoons and we have an obvious promotional push going on. But I simply must recommend the (possibly) more mundane (no comics) but nonetheless excellent David H. Freedman book: "Wrong: Why experts keep failing us--and how to know when not to trust them Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more" https://www.amazon.com/Wrong-us-Scientists-relationship-consultants/dp/0316023787 https://www.amazon.com/Wrong-us-Scientists-relationship-cons... which begins with an interview with John Ioannidis and goes on to discuss in detail why so many academic (and expert) publications are wrong and how they got that way.
- annoyingnoob 6y agoAll of these problems plus low pay, why would anyone go into academia?
- annoyingnoob 6y agoYou can down vote me but I asked a serious question. There are certainly science careers that are not in academia. Am I missing something? Is there great pay before you've taken a lot of time to move up the ladder, assuming you are fortunate enough? Help me understand your reasoning rather than just down voting me.
- pvaldes 6y agoLOL, The annelida part killed me
- kerkeslager 6y agoOne big problem is that journals agree to publish studies after they are completed, which means that publication in prestigious journals is based on the novelty of the result rather than on the validity of the study/experiment. It's an absolute fundamental of science that you go into it with an open mind, admitting that you do not know what the result will be. A good study/experiment is not one which produces an interesting result, it's one which is properly designed to answer the question it's asking. Evaluating the quality of studies/experiments based on their results is an anti-scientific practice which should be excised like the cancerous tumor it is. The best way to do that is to accept and commit to publish studies/experiments based solely on their design, before the experiment/study has actually been performed.
- ylem 6y agoThis is actually being done in some journals--with registration of studies. I have arguments at times with people about Scientific Reports--I think that as long as the results are technically correct, I think we really need to encourage people to publish "boring" results. I have done this as community service, but it takes a lot of time and effort--even more so if you are correcting a previous boring with result with another boring result.
- kerkeslager 6y agoI don't think "encouraging" is enough. It needs to be built into the fabric of how science is performed.
- Seabiscuit 6y agoIsn't this the idea behind Registered Reports? (https://www.cos.io/our-services/registered-reports https://www.cos.io/our-services/registered-reports / https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/registered-reports-a-step-change-in-scientific-publishing https://www.elsevier.com/reviewers-update/story/innovation-i...)
- renewiltord 6y agoOh I like that. With hypothesis pre-registration, journals could commit based on the hypothesis (and perhaps methods) rather than on the result.
- csours 6y agoWhat you like to see at the top of any article covering the press release of a study? Something like an infobox with P-factor, whether it was pre-registered, sample size, funding organization, double-blind, etc? This comic tackles the Academia side of things, but a lot of that motivation comes from press coverage. If the press has better capabilities to be critical of bad studies, Academia will give less credence to the same.
- kkotak 6y agoI think all scientific papers should be written in this format.
- t0mbstone 6y agoI can't help but wonder how many of the issues with scientific papers couldn't be solved with technology. The notion that science is so deeply rooted in antiquated concepts like paper journals and academic constructs like tenure is absurd when the internet has been around for as long as it has. For example, imagine if scientific papers were voted up or down by a community, kind of like stack overflow. Or imagine if scientific papers had to publish all of their source materials and instructions for replicating the experiment, and there was a system for tracking and showing whether or not the experiment had been validated or disproven? What if you "game-ified" scientific papers and gave people points for publishing, but also gave people twice as many points for disproving a paper? Imagine if we had a platform for tracking scientific theories and experiments that was a combination of democratic/meritocratic administration (like wikipedia), change logging/tracking (like github), and reputation management (like stack overflow)...
- fritzo 6y agoparaphrasing to emphasize irony: "Science should be based on solid data: published, auditable, peer-reviewed numbers. Data is good, data is objective, data is truth. "Academic hiring is broken. We can't base academic hiring on numbers because people game the numbers. In academic hiring we need to be subjective, to evaluate the intrinsic merit of each researcher. Data is corrupt, data isn't sufficiently subjective, data is flawed.
- amatic 6y agoI think the core problem is our ignorance of psychology - we don't know how humans really work, what makes us tick, what are the 'incentives' that should be put in system design to move toward better science, and whether 'incentives' are even a good conceptualization of human motivation. We will not fix science until we understand what makes scientists behave as they do, and until we figure out how to design systems for humans. Though, Maybe we stumble upon a better system via blind variation in system properties and selective retention, based on some novel metric. Scientific psychology is rather weak in explaining and predicting how humans will behave.
- gentleman11 6y agoThe fixes they propose are good ones, but aren’t grounded in reality. They ignore how we got here in the first place. Citations matter as a proxy for importance. Negative studies are inherently uninteresting. Companies fund groundbreaking work because they want to be associated with a breakthrough. Scientists publish in bad journals because their careers depend on it - it’s an entire lifetimes work to get tenure and research grants. They can’t just throw that away. Journals publish lousy studies because they don’t have enough good ones - the journals will not self destruct in order to slim down for us. To fix the system will take a more honest look at the incentives of the people/institutions who create the incentives, and so on.
- drummer 6y agoNothing proves that comic more than the current covid-19 'pandemic' which is largely based on fear and BS (bad science). The doctors and scientists that actually make sense get censored into obscurity while sensational and fear agenda promoting info gets published.
- goatinaboat 6y agoNobody in the world gets to do the “fun” part of their job more than a fraction of their time. I don’t think scientists are uniquely hard done by here. They are enormously more privileged than the vast majority of people, being funded to do something purely speculative, it’s not too much to ask them to publish it so others can benefit from that spending too - which largely comes from taxpayers doing less fulfilling jobs.
- Vinnl 6y agoIt's an excellent analysis of the fundamental problems in academia, but "people should just act against their incentives" isn't really a solution. It really is the incentives themselves that are the problem: just looking at number of publications and citations (or even: citations of articles in the journals that your articles happen to be published in as well) when determine who to fund or hire. The problem there is that we have those metrics, are relatively quick and easy to obtain, which are accepted because they are what's been used so far - even though plenty of research has pointed out their flaws yet [1]. And anything new that is proposed as a replacement of those metrics (whether other metrics, or other systems of evaluation) is dismissed for not being proven to live up to a standard that the currently used methods do not either, or for not being available quickly or easily enough. (Which is reasonable - e.g. it's not viable to read and properly evaluate all research of your applicants.) (Disclosure: I do volunteer for a project, https://plaudit.pub https://plaudit.pub, that tries to offer an alternative nevertheless.) [1] https://medium.com/flockademic/the-ridiculous-number-that-can-make-or-break-academic-careers-704e00ae070a https://medium.com/flockademic/the-ridiculous-number-that-ca...