2 ms·

Why would you even do this? On the days where I was still energetic enough to try and debunk some of the pseudosciences, I quickly discovered that one of the p

Why would you even do this? On the days where I was still energetic enough to try and debunk some of the pseudosciences, I quickly discovered that one of the primary rules is absolutely never to trust one single article, however "scientific", and however reputable the publisher of the journal/conference is. Otherwise one quickly becomes overwhelmed by the amount of crap published by otherwise "respectable" publishers (e.g. Elsevier's Journal of Homeopathy). Let's not even get into non-reviewed articles... The only things that _may_ have any value whatsoever are the ones published in utmost respectable _journals_ themselves (e.g. Nature, but absolutely not on any of their spinoffs), and/or when there are very big labs/organizations in the authors list. Very big. I mean usually national pharmaceutical associations (obviously not from all states) or huge corporations (who will generally avoid putting their name on the line either way). Meta-analyses most importantly do NOT have any intrinsinc value either _just because_ they are meta-analyses. By reading bioXiv directly, you are most likely doing yourself a disservice. Unless you are a state agency or huge corporation having huge resources backing your which allows you to internally replicate whatever they claim.

whatshisface 7y ago
You're going too far in dismissing the value of individual scientific papers. If you read the methods and analysis yourself you can figure out how good their results are. If you read a lot of papers or read several meta-analyses you can form a opinion about what the consensus is. All of these options are more work than reading one abstract and calling it a day but that doesn't mean scientific papers are worthless. If scientific papers did not contain any information then scientists would not read them.
- AshamedCaptain 7y ago
  No, you cannot figure out by yourself "how good the results are". That is the one of the critical points. The paper might even pass peer review and get accepted -- which, assuming a reasonable quality journal, means that the methods and analysis they claim make sense. But that is just the _very minimum_ you can expect in an article. Incredibly enough, by itself, that is already a huge filter. But sadly that is nowhere near enough to give any value whatsoever to the article. i.e. the article can be perfect and still just wrong. Even assuming good faith from the authors, they could just be the lucky 5% who got a positive out of an impossible thing and decided to publish. In a world with a general pressure to publish positive results and specially in a highly-hyped topic such as this one, you have a very high chance of seeing such a result. At the end of the day this is the only argument I have left to debunk some homeopathy papers. Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. "Mortals" are not and reading these articles is a waste of time and potentially a disservice. There are tens of journals claiming _utter non-sense_ being published every single month. Using un-reviewed papers to estimate "consensus" is also pointless. Using publication numbers / citations / impact factors is slightly better but also extremely dangerous. If anything pseudo-science chaps are good at is at publishing large numbers of astonishing crap. The sad truth is that filtering good science vs not is out of reach for the average layman, which leaves you with unscientific stuff such as "reputation" as your only tool.
  - whatshisface 7y ago
    I think you might want to qualify that by specific fields. For example if the paper you're reading is math or theoretical physics, then you actually do have all of the same capabilities as the author to verify it. As for publication bias, some studies are pre-registered. >Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. It is typical for a scientist to read hundreds and hundreds of papers every year, or sometimes even every month. Are you suggesting that they reproduce every one of them? You may be dismayed to find out that typically, they would reproduce none of them, and when the system is working its best they might reproduce one of them. Under usual circumstances, you and they have exactly the same replication ability: the ability to search the literature for a replication.
mikk14 7y ago
While the general sentiment and advice you give is mostly correct, I feel like adding my two cents from my experience -- as a person who occasionally publishes in these sorts of journals. In my opinion, you have the Science/Nature thing vs their spinoff exactly the other way around. Science/Nature only publish flashy things more for their "memeness" value than their actual scientific content. The first driver leading you to those journal is how much the paper can be talked about and picked up by other media outlets. In fact, the most important thing you need to publish in Science/Nature is NOT a good experiment/result, but being a coauthor of somebody who already has a lot of Science/Nature papers. Their spinoffs, on the other hand, being less driven by popularity, often offer more solid content.
0003 7y ago
The last paragraph is key and should be emphasized.

whatshisface 7y ago

You're going too far in dismissing the value of individual scientific papers. If you read the methods and analysis yourself you can figure out how good their results are. If you read a lot of papers or read several meta-analyses you can form a opinion about what the consensus is. All of these options are more work than reading one abstract and calling it a day but that doesn't mean scientific papers are worthless. If scientific papers did not contain any information then scientists would not read them.

AshamedCaptain 7y ago
No, you cannot figure out by yourself "how good the results are". That is the one of the critical points. The paper might even pass peer review and get accepted -- which, assuming a reasonable quality journal, means that the methods and analysis they claim make sense. But that is just the _very minimum_ you can expect in an article. Incredibly enough, by itself, that is already a huge filter. But sadly that is nowhere near enough to give any value whatsoever to the article. i.e. the article can be perfect and still just wrong. Even assuming good faith from the authors, they could just be the lucky 5% who got a positive out of an impossible thing and decided to publish. In a world with a general pressure to publish positive results and specially in a highly-hyped topic such as this one, you have a very high chance of seeing such a result. At the end of the day this is the only argument I have left to debunk some homeopathy papers. Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. "Mortals" are not and reading these articles is a waste of time and potentially a disservice. There are tens of journals claiming _utter non-sense_ being published every single month. Using un-reviewed papers to estimate "consensus" is also pointless. Using publication numbers / citations / impact factors is slightly better but also extremely dangerous. If anything pseudo-science chaps are good at is at publishing large numbers of astonishing crap. The sad truth is that filtering good science vs not is out of reach for the average layman, which leaves you with unscientific stuff such as "reputation" as your only tool.
- whatshisface 7y ago
  I think you might want to qualify that by specific fields. For example if the paper you're reading is math or theoretical physics, then you actually do have all of the same capabilities as the author to verify it. As for publication bias, some studies are pre-registered. >Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. It is typical for a scientist to read hundreds and hundreds of papers every year, or sometimes even every month. Are you suggesting that they reproduce every one of them? You may be dismayed to find out that typically, they would reproduce none of them, and when the system is working its best they might reproduce one of them. Under usual circumstances, you and they have exactly the same replication ability: the ability to search the literature for a replication.

mikk14 7y ago

While the general sentiment and advice you give is mostly correct, I feel like adding my two cents from my experience -- as a person who occasionally publishes in these sorts of journals. In my opinion, you have the Science/Nature thing vs their spinoff exactly the other way around. Science/Nature only publish flashy things more for their "memeness" value than their actual scientific content. The first driver leading you to those journal is how much the paper can be talked about and picked up by other media outlets. In fact, the most important thing you need to publish in Science/Nature is NOT a good experiment/result, but being a coauthor of somebody who already has a lot of Science/Nature papers. Their spinoffs, on the other hand, being less driven by popularity, often offer more solid content.

0003 7y ago

The last paragraph is key and should be emphasized.