12 ms·

Yes. And let's not get started on that ML Quantum Wormhole bullshit... We've taken this all too far. It is bad enough to lie to the masses in Pop-Sci articles.

Yes. And let's not get started on that ML Quantum Wormhole bullshit... We've taken this all too far. It is bad enough to lie to the masses in Pop-Sci articles. But we're straight up doing it in top tier journals. Some are good faith mistakes, but a lot more often they seem like due diligence just wasn't ever done. Both by researchers and reviewers. I at least have to thank the journals. I've hated them for a long time and wanted to see their end. Free up publishing and bullshit novelty and narrowing of research. I just never thought they'd be the ones to put the knife through their own heart. But I'm still not happy about that tbh. The only result of this is that the public grows to distrust science more and more. In a time where we need that trust more than ever. We can't expect the public to differentiate nuanced takes about internal quibbling. And we sure as hell shouldn't be giving ammunition to the anti-science crowds, like junk science does...

toofy 1y ago
this seems strange to me, shouldn’t we expect a high quality journal to retract often as we gather more information? obviously this is hyperbole of two extremes, but i certainly trust a journal far more if it actively and loudly looks to correct mistakes over one that never corrects anything or buries its retractions. a rather important piece of science is correcting mistakes by gathering and testing new information. we should absolutely be applauding when a journal loudly and proactively says “oh, it turns out we were wrong when we declared burying a chestnut under the oak tree on the third thursday of a full moon would cure your brothers infected toenail.”
- Ekaros 1y ago
  I think it might come to understanding what is "high quality" journal. Maybe such journal should be focused on much more proven and mature things. Where there would be lot less retractions as they have more mature and thus more proven information. But I think problem is what is seen is "high quality" == "high impact". Which means that prestige and visibility is important things. Which likely lowers the threshold quite a lot as being first to publish possibly valid something is seen as important.
- godelski 1y ago
  > shouldn’t we expect a high quality journal to retract often as we gather more information? This is complicated, and kinda sad tbh. But no. You need to carefully think about what "high quality journal" means. Typically it is based on something called Impact Factor[0]. Impact factor is judged by the number of citations a journal has received in the last 2 years. It sounds good on paper, but I think if you think about it for a second you'll notice there's a positive feedback loop. There's also no incentive that it is actually correct. For example, a false paper can often get cited far more than a true paper. This is because when you write the academic version of "XYZ is a fucking idiot, and here's why" you cite their paper. It's good to put their bullshit down, but it can also just end up being Streisand effect-like. Journal is happy with its citations. Both people published in them. They benefit from both directions. You keep the bad paper up for the record and because as long as the authors were actually acting in good faith, you don't actually want to take it down. The problem is... how do you know? Another weird factor used is Acceptance Rates. This again sounds nice at first. You don't want a journal publishing just anything, right?[1] The problem comes when these actually become targets (which they are). Many of the ML conferences target about 25% acceptance rate[2]. It fluctuates year to year. It should, right? Some years are just better science than other years. Good paper hits that changes things and the next year should have a boom! But that's not the level of fluctuation we're talking about. If you look at the actual number of papers accepted in that repo you'll see a disproportionate number of accepted papers ending in a 0 or 5. Then you see the 1 and 6, which is a paper being squeezed in, often for political reasons. Here, I did the first 2 tables for you. You'll see that has a very disproportionate ending of 1 and 6 and CV loves 0,1,3 These numbers should convince you that this is not a random process, though they should not convince you it is all funny business (much harder to prove). But it is at least enough to be suspicious and encourage you to dig in more. There's a lot that's fucked up about the publishing system and academia. Lots of politics, lots of restricted research directions, lots of stupid. But also don't confuse this for people acting in bad faith or lying. Sure, that happens. But most people are trying to do good and very few people in academia are blatantly publishing bullshit. It's just that everything gets political. And by political I don't mean government politics, I mean the same bullshit office politics. We're not immune from that same bullshit and it happens for exactly the same reasons. It just gets messier because if you think it is hard to measure the output of an employee, try to measure the output of people who's entire job it is to create things that no one has ever thought of before. It's sure going to look like they're doing a whole lot of nothing. So I'll just leave you with this (it'll explain [1]) As a working scientist, Mervin Kelly (Director of Bell Labs (1925-1959)) understood the golden rule, "How do you manage genius? You don't." https://1517.substack.com/p/why-bell-labs-worked There's more complexity like how we aren't good at pushing out frauds and stuff but if you want that I'll save it for another comment. [0] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [1] Actually I do. As long as it isn't obviously wrong, plagiarized, or falsified, then I want that published. You did work, you communicated it, now I want it to get out into the public so that it can be peer reviewed. I don't mean a journal's laughable version of peer review (3-4 unpaid people that don't study your niche and are more concerned with if it is "novel" or "impactful" quickly reading your paper and you're one of 4 on their desk they need to do this week. It's incredibly subjective and high impact papers (like Nobel Price winning papers) routinely get rejected). Peer review is the process of other researchers replicating your work, building on it, and/or countering it. Those are just new papers... [2] https://github.com/lixin4ever/Conference-Acceptance-Rate https://github.com/lixin4ever/Conference-Acceptance-Rate [3] https://imgur.com/a/YefOcuA https://imgur.com/a/YefOcuA
somenameforme 1y ago
Almost nobody is "anti-science". The source of that labeling and division came from appeals to authority. You must do or believe this because it's "the science." If you don't, or you disagree, then you are anti-science. It has nothing to do with science, but rather people not finding that a sufficient justification for unpopular actions. For instance it's 100% certain that banning sugary drinks would dramatically improve public health, reduce healthcare costs, increase life expectancy, and just generally make society better in every single way. So should we ban sugary drinks? It'd be akin to me trying to claim that if you say no then you're anti-science, anti-health, or whatever else. It's just a dumb, divisive, and meaningless label - exactly the sort politicians love so much now a days. Of course there's some irony in that it will become a self fulfilling prophecy. The more unpopular things done in the name of 'the science', the more negative public sentiment to 'the science' will become. Probably somewhat similar to how societies gradually became secular over time, as it became quite clear that actions done in the name of God were often not exactly pious.
- tensor 1y ago
  Yes, there are a lot of people who are anti-science. As in they do not believe the scientific method is a good way to find truth. There are people today who are rejecting very basic science that was accepted over a century ago.
  - somenameforme 1y ago
    No, there aren't. Most people don't realize when they're being trolled by things like e.g. flat earth types. Go read one of the groups, it's a trolling meme largely turned into something by the internet + media. Thanks to social media even if let's say 0.1% of English speakers believe (or pretend to believe) something, that'd be 1.6 million people, so you can get a false impression, especially when the media takes the trolling and then amplifies it with a straight face largely to amplify these dumb and farcical divides.
    - tbrownaw 1y ago
      I've seen science (the process, not just the results of that process) denounced as an aspect or tool of Western colonialism / imperialism. And there's that related "indigenous ways of knowing" thing that Canada has going on. --- Also the flat earth people actually aren't trying to argue against science (the process). They're arguing that everyone except them made either observational errors or reasoning errors.

toofy 1y ago

this seems strange to me, shouldn’t we expect a high quality journal to retract often as we gather more information? obviously this is hyperbole of two extremes, but i certainly trust a journal far more if it actively and loudly looks to correct mistakes over one that never corrects anything or buries its retractions. a rather important piece of science is correcting mistakes by gathering and testing new information. we should absolutely be applauding when a journal loudly and proactively says “oh, it turns out we were wrong when we declared burying a chestnut under the oak tree on the third thursday of a full moon would cure your brothers infected toenail.”

Ekaros 1y ago
I think it might come to understanding what is "high quality" journal. Maybe such journal should be focused on much more proven and mature things. Where there would be lot less retractions as they have more mature and thus more proven information. But I think problem is what is seen is "high quality" == "high impact". Which means that prestige and visibility is important things. Which likely lowers the threshold quite a lot as being first to publish possibly valid something is seen as important.
godelski 1y ago
> shouldn’t we expect a high quality journal to retract often as we gather more information? This is complicated, and kinda sad tbh. But no. You need to carefully think about what "high quality journal" means. Typically it is based on something called Impact Factor[0]. Impact factor is judged by the number of citations a journal has received in the last 2 years. It sounds good on paper, but I think if you think about it for a second you'll notice there's a positive feedback loop. There's also no incentive that it is actually correct. For example, a false paper can often get cited far more than a true paper. This is because when you write the academic version of "XYZ is a fucking idiot, and here's why" you cite their paper. It's good to put their bullshit down, but it can also just end up being Streisand effect-like. Journal is happy with its citations. Both people published in them. They benefit from both directions. You keep the bad paper up for the record and because as long as the authors were actually acting in good faith, you don't actually want to take it down. The problem is... how do you know? Another weird factor used is Acceptance Rates. This again sounds nice at first. You don't want a journal publishing just anything, right?[1] The problem comes when these actually become targets (which they are). Many of the ML conferences target about 25% acceptance rate[2]. It fluctuates year to year. It should, right? Some years are just better science than other years. Good paper hits that changes things and the next year should have a boom! But that's not the level of fluctuation we're talking about. If you look at the actual number of papers accepted in that repo you'll see a disproportionate number of accepted papers ending in a 0 or 5. Then you see the 1 and 6, which is a paper being squeezed in, often for political reasons. Here, I did the first 2 tables for you. You'll see that has a very disproportionate ending of 1 and 6 and CV loves 0,1,3 These numbers should convince you that this is not a random process, though they should not convince you it is all funny business (much harder to prove). But it is at least enough to be suspicious and encourage you to dig in more. There's a lot that's fucked up about the publishing system and academia. Lots of politics, lots of restricted research directions, lots of stupid. But also don't confuse this for people acting in bad faith or lying. Sure, that happens. But most people are trying to do good and very few people in academia are blatantly publishing bullshit. It's just that everything gets political. And by political I don't mean government politics, I mean the same bullshit office politics. We're not immune from that same bullshit and it happens for exactly the same reasons. It just gets messier because if you think it is hard to measure the output of an employee, try to measure the output of people who's entire job it is to create things that no one has ever thought of before. It's sure going to look like they're doing a whole lot of nothing. So I'll just leave you with this (it'll explain [1]) As a working scientist, Mervin Kelly (Director of Bell Labs (1925-1959)) understood the golden rule, "How do you manage genius? You don't." https://1517.substack.com/p/why-bell-labs-worked There's more complexity like how we aren't good at pushing out frauds and stuff but if you want that I'll save it for another comment. [0] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [1] Actually I do. As long as it isn't obviously wrong, plagiarized, or falsified, then I want that published. You did work, you communicated it, now I want it to get out into the public so that it can be peer reviewed. I don't mean a journal's laughable version of peer review (3-4 unpaid people that don't study your niche and are more concerned with if it is "novel" or "impactful" quickly reading your paper and you're one of 4 on their desk they need to do this week. It's incredibly subjective and high impact papers (like Nobel Price winning papers) routinely get rejected). Peer review is the process of other researchers replicating your work, building on it, and/or countering it. Those are just new papers... [2] https://github.com/lixin4ever/Conference-Acceptance-Rate https://github.com/lixin4ever/Conference-Acceptance-Rate [3] https://imgur.com/a/YefOcuA https://imgur.com/a/YefOcuA

somenameforme 1y ago

Almost nobody is "anti-science". The source of that labeling and division came from appeals to authority. You must do or believe this because it's "the science." If you don't, or you disagree, then you are anti-science. It has nothing to do with science, but rather people not finding that a sufficient justification for unpopular actions. For instance it's 100% certain that banning sugary drinks would dramatically improve public health, reduce healthcare costs, increase life expectancy, and just generally make society better in every single way. So should we ban sugary drinks? It'd be akin to me trying to claim that if you say no then you're anti-science, anti-health, or whatever else. It's just a dumb, divisive, and meaningless label - exactly the sort politicians love so much now a days. Of course there's some irony in that it will become a self fulfilling prophecy. The more unpopular things done in the name of 'the science', the more negative public sentiment to 'the science' will become. Probably somewhat similar to how societies gradually became secular over time, as it became quite clear that actions done in the name of God were often not exactly pious.

tensor 1y ago
Yes, there are a lot of people who are anti-science. As in they do not believe the scientific method is a good way to find truth. There are people today who are rejecting very basic science that was accepted over a century ago.
- somenameforme 1y ago
  No, there aren't. Most people don't realize when they're being trolled by things like e.g. flat earth types. Go read one of the groups, it's a trolling meme largely turned into something by the internet + media. Thanks to social media even if let's say 0.1% of English speakers believe (or pretend to believe) something, that'd be 1.6 million people, so you can get a false impression, especially when the media takes the trolling and then amplifies it with a straight face largely to amplify these dumb and farcical divides.
  - tbrownaw 1y ago
    I've seen science (the process, not just the results of that process) denounced as an aspect or tool of Western colonialism / imperialism. And there's that related "indigenous ways of knowing" thing that Canada has going on. --- Also the flat earth people actually aren't trying to argue against science (the process). They're arguing that everyone except them made either observational errors or reasoning errors.