6 ms·

It’s like fake news is taking in science now. Saying any stupid thing will attract much more view and « likes » than those debunking them. Except that we can’t

It’s like fake news is taking in science now. Saying any stupid thing will attract much more view and « likes » than those debunking them. Except that we can’t compare twitter to nature journal. Science is supposed to be immune to these kind of bullshit thanks to reputed journals and pair reviewing, blocking a publication before it does any harm. Was that a failure of nature ?

godelski 1y ago
Yes. And let's not get started on that ML Quantum Wormhole bullshit... We've taken this all too far. It is bad enough to lie to the masses in Pop-Sci articles. But we're straight up doing it in top tier journals. Some are good faith mistakes, but a lot more often they seem like due diligence just wasn't ever done. Both by researchers and reviewers. I at least have to thank the journals. I've hated them for a long time and wanted to see their end. Free up publishing and bullshit novelty and narrowing of research. I just never thought they'd be the ones to put the knife through their own heart. But I'm still not happy about that tbh. The only result of this is that the public grows to distrust science more and more. In a time where we need that trust more than ever. We can't expect the public to differentiate nuanced takes about internal quibbling. And we sure as hell shouldn't be giving ammunition to the anti-science crowds, like junk science does...
- toofy 1y ago
  this seems strange to me, shouldn’t we expect a high quality journal to retract often as we gather more information? obviously this is hyperbole of two extremes, but i certainly trust a journal far more if it actively and loudly looks to correct mistakes over one that never corrects anything or buries its retractions. a rather important piece of science is correcting mistakes by gathering and testing new information. we should absolutely be applauding when a journal loudly and proactively says “oh, it turns out we were wrong when we declared burying a chestnut under the oak tree on the third thursday of a full moon would cure your brothers infected toenail.”
  - Ekaros 1y ago
    I think it might come to understanding what is "high quality" journal. Maybe such journal should be focused on much more proven and mature things. Where there would be lot less retractions as they have more mature and thus more proven information. But I think problem is what is seen is "high quality" == "high impact". Which means that prestige and visibility is important things. Which likely lowers the threshold quite a lot as being first to publish possibly valid something is seen as important.
  - godelski 1y ago
    > shouldn’t we expect a high quality journal to retract often as we gather more information? This is complicated, and kinda sad tbh. But no. You need to carefully think about what "high quality journal" means. Typically it is based on something called Impact Factor[0]. Impact factor is judged by the number of citations a journal has received in the last 2 years. It sounds good on paper, but I think if you think about it for a second you'll notice there's a positive feedback loop. There's also no incentive that it is actually correct. For example, a false paper can often get cited far more than a true paper. This is because when you write the academic version of "XYZ is a fucking idiot, and here's why" you cite their paper. It's good to put their bullshit down, but it can also just end up being Streisand effect-like. Journal is happy with its citations. Both people published in them. They benefit from both directions. You keep the bad paper up for the record and because as long as the authors were actually acting in good faith, you don't actually want to take it down. The problem is... how do you know? Another weird factor used is Acceptance Rates. This again sounds nice at first. You don't want a journal publishing just anything, right?[1] The problem comes when these actually become targets (which they are). Many of the ML conferences target about 25% acceptance rate[2]. It fluctuates year to year. It should, right? Some years are just better science than other years. Good paper hits that changes things and the next year should have a boom! But that's not the level of fluctuation we're talking about. If you look at the actual number of papers accepted in that repo you'll see a disproportionate number of accepted papers ending in a 0 or 5. Then you see the 1 and 6, which is a paper being squeezed in, often for political reasons. Here, I did the first 2 tables for you. You'll see that has a very disproportionate ending of 1 and 6 and CV loves 0,1,3 These numbers should convince you that this is not a random process, though they should not convince you it is all funny business (much harder to prove). But it is at least enough to be suspicious and encourage you to dig in more. There's a lot that's fucked up about the publishing system and academia. Lots of politics, lots of restricted research directions, lots of stupid. But also don't confuse this for people acting in bad faith or lying. Sure, that happens. But most people are trying to do good and very few people in academia are blatantly publishing bullshit. It's just that everything gets political. And by political I don't mean government politics, I mean the same bullshit office politics. We're not immune from that same bullshit and it happens for exactly the same reasons. It just gets messier because if you think it is hard to measure the output of an employee, try to measure the output of people who's entire job it is to create things that no one has ever thought of before. It's sure going to look like they're doing a whole lot of nothing. So I'll just leave you with this (it'll explain [1]) As a working scientist, Mervin Kelly (Director of Bell Labs (1925-1959)) understood the golden rule, "How do you manage genius? You don't." https://1517.substack.com/p/why-bell-labs-worked There's more complexity like how we aren't good at pushing out frauds and stuff but if you want that I'll save it for another comment. [0] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [1] Actually I do. As long as it isn't obviously wrong, plagiarized, or falsified, then I want that published. You did work, you communicated it, now I want it to get out into the public so that it can be peer reviewed. I don't mean a journal's laughable version of peer review (3-4 unpaid people that don't study your niche and are more concerned with if it is "novel" or "impactful" quickly reading your paper and you're one of 4 on their desk they need to do this week. It's incredibly subjective and high impact papers (like Nobel Price winning papers) routinely get rejected). Peer review is the process of other researchers replicating your work, building on it, and/or countering it. Those are just new papers... [2] https://github.com/lixin4ever/Conference-Acceptance-Rate https://github.com/lixin4ever/Conference-Acceptance-Rate [3] https://imgur.com/a/YefOcuA https://imgur.com/a/YefOcuA
- somenameforme 1y ago
  Almost nobody is "anti-science". The source of that labeling and division came from appeals to authority. You must do or believe this because it's "the science." If you don't, or you disagree, then you are anti-science. It has nothing to do with science, but rather people not finding that a sufficient justification for unpopular actions. For instance it's 100% certain that banning sugary drinks would dramatically improve public health, reduce healthcare costs, increase life expectancy, and just generally make society better in every single way. So should we ban sugary drinks? It'd be akin to me trying to claim that if you say no then you're anti-science, anti-health, or whatever else. It's just a dumb, divisive, and meaningless label - exactly the sort politicians love so much now a days. Of course there's some irony in that it will become a self fulfilling prophecy. The more unpopular things done in the name of 'the science', the more negative public sentiment to 'the science' will become. Probably somewhat similar to how societies gradually became secular over time, as it became quite clear that actions done in the name of God were often not exactly pious.
  - tensor 1y ago
    Yes, there are a lot of people who are anti-science. As in they do not believe the scientific method is a good way to find truth. There are people today who are rejecting very basic science that was accepted over a century ago.
    - somenameforme 1y ago
      No, there aren't. Most people don't realize when they're being trolled by things like e.g. flat earth types. Go read one of the groups, it's a trolling meme largely turned into something by the internet + media. Thanks to social media even if let's say 0.1% of English speakers believe (or pretend to believe) something, that'd be 1.6 million people, so you can get a false impression, especially when the media takes the trolling and then amplifies it with a straight face largely to amplify these dumb and farcical divides.
      - tbrownaw 1y ago
        I've seen science (the process, not just the results of that process) denounced as an aspect or tool of Western colonialism / imperialism. And there's that related "indigenous ways of knowing" thing that Canada has going on. --- Also the flat earth people actually aren't trying to argue against science (the process). They're arguing that everyone except them made either observational errors or reasoning errors.
lamename 1y ago
The Bullshit asymmetry principle comes to mind https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law
lamename 1y ago
Have you seen the statistics about high impact journals having higher retraction/unverified rates on papers? The root causes can be argued...but keep that in mind. No single paper is proof. Bodies of work across many labs, independent verification, etc is the actual gold standard.
- somenameforme 1y ago
  This is something I think many people don't appreciate. A perfect example in practice is the Journal of Personality and Social Psychology. It's one of the leading and highest impact journals in psychology. A quick search for that name will show it as the source for endless 'news' articles from sites like the NYTimes [1]. And that journal has a 23% replication success rate [2] meaning there's about an 80% chance that anything you read in the journal, and consequently from the numerous sites that love to quote it, is wrong. [1] - https://search.brave.com/search?q=site%3Anytimes.com+Journal+of+Personality+and+Social+Psychology https://search.brave.com/search?q=site%3Anytimes.com+Journal... [2] - https://en.wikipedia.org/wiki/Replication_crisis#In_psychology https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo...
- vkou 1y ago
  The purpose of peer review is to check for methodological errors, not to replicate the experiment. With a few exceptions, it can't catch many categories of serious errors. > higher retraction/unverified Scientific consensus doesn't advance because a single new ground-breaking claim is made in a prestigious journal. It advances when enough other scientists have built on top of that work. The current state of science is not 'bleeding edge stuff published in a journal last week'. That bleeding edge stuff might become part of scientific consensus in a month, or year or three, or five - when enough other people build on that work. Anybody who actually does science understands this. Unfortunately, people with poor media literacy who only read the headlines don't understand this, and assume that the whole process is all a crock.
tbrownaw 1y ago
> It’s like fake news is taking in science now. I didn't think this was new? Like, it's been a few years since that replication crisis things kicked off.
imiric 1y ago
> Science is supposed to be immune to these kind of bullshit You have misplaced confidence in the scientific method. It was never immune to corruption, either by those deliberately manipulating it for their personal gain, or simply due to ignorance and bad methodology. We have examples of both throughout history. In either case, peer review is not infallible. The new problem introduced by modern AI tools is that they drastically lower the skill requirement for anyone remotely capable in the field to generate data that appears correct on the surface, with relatively little effort and very quickly, while errors can only be discovered by actual experts in the field investing considerable amounts of time and resources. In some fields like programming the required resources to review code are relatively minor, but in fields like biology this (from what I've read) is much more difficult and expensive. But, yes, science is being flooded with (m|d)isinformation, just like all our other media channels.
dathinab 1y ago
no it's a long term incoming failure partially due to legacy of science historically being rooted in "it matters more who you (or your parents) are" societies (due to them having had the money in somewhat modern history) (or like some would say the "old white man problem", except it has nothing to do with skin color, or man and only limited to do with old) partially due to how much more "science (output)" is produced today and ways which once worked to have reasonable QA don't work that well in todays scale anymore partially due to how many flows partially due to human nature (as in people tend to care more about "exiting", "visible" things etc.) People have been pushing for change in a lot of ways like: - pushing to make full re-poducability a must have (but that is hard, especially for statistics based things only a few companies can even afford to try to run. But also hard due to it requiring a lot of transparency and open data access, and especially the alter is often very much something many owners of data sets are not okay with. - pushing for more appreciation of null results, or failures. (To be clear I mean both appreciation in form of monetary support and in the traditional sense of the word of people (colleges) appreciating it). - pushing for more verifying of papers by trying to reproduce it (both as in more money/time resources for it and in changing the mind set from it being a daunting unappreciated task to it being a nice thing to do) but to little change happened in the end before modern LLM AI hit the scene and now it has made things so much harder as it's now easy to mass produce slob but reasonable looking (non) sience

godelski 1y ago

Yes. And let's not get started on that ML Quantum Wormhole bullshit... We've taken this all too far. It is bad enough to lie to the masses in Pop-Sci articles. But we're straight up doing it in top tier journals. Some are good faith mistakes, but a lot more often they seem like due diligence just wasn't ever done. Both by researchers and reviewers. I at least have to thank the journals. I've hated them for a long time and wanted to see their end. Free up publishing and bullshit novelty and narrowing of research. I just never thought they'd be the ones to put the knife through their own heart. But I'm still not happy about that tbh. The only result of this is that the public grows to distrust science more and more. In a time where we need that trust more than ever. We can't expect the public to differentiate nuanced takes about internal quibbling. And we sure as hell shouldn't be giving ammunition to the anti-science crowds, like junk science does...

toofy 1y ago
this seems strange to me, shouldn’t we expect a high quality journal to retract often as we gather more information? obviously this is hyperbole of two extremes, but i certainly trust a journal far more if it actively and loudly looks to correct mistakes over one that never corrects anything or buries its retractions. a rather important piece of science is correcting mistakes by gathering and testing new information. we should absolutely be applauding when a journal loudly and proactively says “oh, it turns out we were wrong when we declared burying a chestnut under the oak tree on the third thursday of a full moon would cure your brothers infected toenail.”
- Ekaros 1y ago
  I think it might come to understanding what is "high quality" journal. Maybe such journal should be focused on much more proven and mature things. Where there would be lot less retractions as they have more mature and thus more proven information. But I think problem is what is seen is "high quality" == "high impact". Which means that prestige and visibility is important things. Which likely lowers the threshold quite a lot as being first to publish possibly valid something is seen as important.
- godelski 1y ago
  > shouldn’t we expect a high quality journal to retract often as we gather more information? This is complicated, and kinda sad tbh. But no. You need to carefully think about what "high quality journal" means. Typically it is based on something called Impact Factor[0]. Impact factor is judged by the number of citations a journal has received in the last 2 years. It sounds good on paper, but I think if you think about it for a second you'll notice there's a positive feedback loop. There's also no incentive that it is actually correct. For example, a false paper can often get cited far more than a true paper. This is because when you write the academic version of "XYZ is a fucking idiot, and here's why" you cite their paper. It's good to put their bullshit down, but it can also just end up being Streisand effect-like. Journal is happy with its citations. Both people published in them. They benefit from both directions. You keep the bad paper up for the record and because as long as the authors were actually acting in good faith, you don't actually want to take it down. The problem is... how do you know? Another weird factor used is Acceptance Rates. This again sounds nice at first. You don't want a journal publishing just anything, right?[1] The problem comes when these actually become targets (which they are). Many of the ML conferences target about 25% acceptance rate[2]. It fluctuates year to year. It should, right? Some years are just better science than other years. Good paper hits that changes things and the next year should have a boom! But that's not the level of fluctuation we're talking about. If you look at the actual number of papers accepted in that repo you'll see a disproportionate number of accepted papers ending in a 0 or 5. Then you see the 1 and 6, which is a paper being squeezed in, often for political reasons. Here, I did the first 2 tables for you. You'll see that has a very disproportionate ending of 1 and 6 and CV loves 0,1,3 These numbers should convince you that this is not a random process, though they should not convince you it is all funny business (much harder to prove). But it is at least enough to be suspicious and encourage you to dig in more. There's a lot that's fucked up about the publishing system and academia. Lots of politics, lots of restricted research directions, lots of stupid. But also don't confuse this for people acting in bad faith or lying. Sure, that happens. But most people are trying to do good and very few people in academia are blatantly publishing bullshit. It's just that everything gets political. And by political I don't mean government politics, I mean the same bullshit office politics. We're not immune from that same bullshit and it happens for exactly the same reasons. It just gets messier because if you think it is hard to measure the output of an employee, try to measure the output of people who's entire job it is to create things that no one has ever thought of before. It's sure going to look like they're doing a whole lot of nothing. So I'll just leave you with this (it'll explain [1]) As a working scientist, Mervin Kelly (Director of Bell Labs (1925-1959)) understood the golden rule, "How do you manage genius? You don't." https://1517.substack.com/p/why-bell-labs-worked There's more complexity like how we aren't good at pushing out frauds and stuff but if you want that I'll save it for another comment. [0] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [1] Actually I do. As long as it isn't obviously wrong, plagiarized, or falsified, then I want that published. You did work, you communicated it, now I want it to get out into the public so that it can be peer reviewed. I don't mean a journal's laughable version of peer review (3-4 unpaid people that don't study your niche and are more concerned with if it is "novel" or "impactful" quickly reading your paper and you're one of 4 on their desk they need to do this week. It's incredibly subjective and high impact papers (like Nobel Price winning papers) routinely get rejected). Peer review is the process of other researchers replicating your work, building on it, and/or countering it. Those are just new papers... [2] https://github.com/lixin4ever/Conference-Acceptance-Rate https://github.com/lixin4ever/Conference-Acceptance-Rate [3] https://imgur.com/a/YefOcuA https://imgur.com/a/YefOcuA
somenameforme 1y ago
Almost nobody is "anti-science". The source of that labeling and division came from appeals to authority. You must do or believe this because it's "the science." If you don't, or you disagree, then you are anti-science. It has nothing to do with science, but rather people not finding that a sufficient justification for unpopular actions. For instance it's 100% certain that banning sugary drinks would dramatically improve public health, reduce healthcare costs, increase life expectancy, and just generally make society better in every single way. So should we ban sugary drinks? It'd be akin to me trying to claim that if you say no then you're anti-science, anti-health, or whatever else. It's just a dumb, divisive, and meaningless label - exactly the sort politicians love so much now a days. Of course there's some irony in that it will become a self fulfilling prophecy. The more unpopular things done in the name of 'the science', the more negative public sentiment to 'the science' will become. Probably somewhat similar to how societies gradually became secular over time, as it became quite clear that actions done in the name of God were often not exactly pious.
- tensor 1y ago
  Yes, there are a lot of people who are anti-science. As in they do not believe the scientific method is a good way to find truth. There are people today who are rejecting very basic science that was accepted over a century ago.
  - somenameforme 1y ago
    No, there aren't. Most people don't realize when they're being trolled by things like e.g. flat earth types. Go read one of the groups, it's a trolling meme largely turned into something by the internet + media. Thanks to social media even if let's say 0.1% of English speakers believe (or pretend to believe) something, that'd be 1.6 million people, so you can get a false impression, especially when the media takes the trolling and then amplifies it with a straight face largely to amplify these dumb and farcical divides.
    - tbrownaw 1y ago
      I've seen science (the process, not just the results of that process) denounced as an aspect or tool of Western colonialism / imperialism. And there's that related "indigenous ways of knowing" thing that Canada has going on. --- Also the flat earth people actually aren't trying to argue against science (the process). They're arguing that everyone except them made either observational errors or reasoning errors.

lamename 1y ago

The Bullshit asymmetry principle comes to mind https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law

lamename 1y ago

Have you seen the statistics about high impact journals having higher retraction/unverified rates on papers? The root causes can be argued...but keep that in mind. No single paper is proof. Bodies of work across many labs, independent verification, etc is the actual gold standard.

somenameforme 1y ago
This is something I think many people don't appreciate. A perfect example in practice is the Journal of Personality and Social Psychology. It's one of the leading and highest impact journals in psychology. A quick search for that name will show it as the source for endless 'news' articles from sites like the NYTimes [1]. And that journal has a 23% replication success rate [2] meaning there's about an 80% chance that anything you read in the journal, and consequently from the numerous sites that love to quote it, is wrong. [1] - https://search.brave.com/search?q=site%3Anytimes.com+Journal+of+Personality+and+Social+Psychology https://search.brave.com/search?q=site%3Anytimes.com+Journal... [2] - https://en.wikipedia.org/wiki/Replication_crisis#In_psychology https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo...
vkou 1y ago
The purpose of peer review is to check for methodological errors, not to replicate the experiment. With a few exceptions, it can't catch many categories of serious errors. > higher retraction/unverified Scientific consensus doesn't advance because a single new ground-breaking claim is made in a prestigious journal. It advances when enough other scientists have built on top of that work. The current state of science is not 'bleeding edge stuff published in a journal last week'. That bleeding edge stuff might become part of scientific consensus in a month, or year or three, or five - when enough other people build on that work. Anybody who actually does science understands this. Unfortunately, people with poor media literacy who only read the headlines don't understand this, and assume that the whole process is all a crock.

tbrownaw 1y ago

> It’s like fake news is taking in science now. I didn't think this was new? Like, it's been a few years since that replication crisis things kicked off.

imiric 1y ago

> Science is supposed to be immune to these kind of bullshit You have misplaced confidence in the scientific method. It was never immune to corruption, either by those deliberately manipulating it for their personal gain, or simply due to ignorance and bad methodology. We have examples of both throughout history. In either case, peer review is not infallible. The new problem introduced by modern AI tools is that they drastically lower the skill requirement for anyone remotely capable in the field to generate data that appears correct on the surface, with relatively little effort and very quickly, while errors can only be discovered by actual experts in the field investing considerable amounts of time and resources. In some fields like programming the required resources to review code are relatively minor, but in fields like biology this (from what I've read) is much more difficult and expensive. But, yes, science is being flooded with (m|d)isinformation, just like all our other media channels.

dathinab 1y ago

no it's a long term incoming failure partially due to legacy of science historically being rooted in "it matters more who you (or your parents) are" societies (due to them having had the money in somewhat modern history) (or like some would say the "old white man problem", except it has nothing to do with skin color, or man and only limited to do with old) partially due to how much more "science (output)" is produced today and ways which once worked to have reasonable QA don't work that well in todays scale anymore partially due to how many flows partially due to human nature (as in people tend to care more about "exiting", "visible" things etc.) People have been pushing for change in a lot of ways like: - pushing to make full re-poducability a must have (but that is hard, especially for statistics based things only a few companies can even afford to try to run. But also hard due to it requiring a lot of transparency and open data access, and especially the alter is often very much something many owners of data sets are not okay with. - pushing for more appreciation of null results, or failures. (To be clear I mean both appreciation in form of monetary support and in the traditional sense of the word of people (colleges) appreciating it). - pushing for more verifying of papers by trying to reproduce it (both as in more money/time resources for it and in changing the mind set from it being a daunting unappreciated task to it being a nice thing to do) but to little change happened in the end before modern LLM AI hit the scene and now it has made things so much harder as it's now easy to mass produce slob but reasonable looking (non) sience