4 ms·

No, you cannot figure out by yourself "how good the results are". That is the one of the critical points. The paper might even pass peer review and get accepted

No, you cannot figure out by yourself "how good the results are". That is the one of the critical points. The paper might even pass peer review and get accepted -- which, assuming a reasonable quality journal, means that the methods and analysis they claim make sense. But that is just the _very minimum_ you can expect in an article. Incredibly enough, by itself, that is already a huge filter. But sadly that is nowhere near enough to give any value whatsoever to the article. i.e. the article can be perfect and still just wrong. Even assuming good faith from the authors, they could just be the lucky 5% who got a positive out of an impossible thing and decided to publish. In a world with a general pressure to publish positive results and specially in a highly-hyped topic such as this one, you have a very high chance of seeing such a result. At the end of the day this is the only argument I have left to debunk some homeopathy papers. Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. "Mortals" are not and reading these articles is a waste of time and potentially a disservice. There are tens of journals claiming _utter non-sense_ being published every single month. Using un-reviewed papers to estimate "consensus" is also pointless. Using publication numbers / citations / impact factors is slightly better but also extremely dangerous. If anything pseudo-science chaps are good at is at publishing large numbers of astonishing crap. The sad truth is that filtering good science vs not is out of reach for the average layman, which leaves you with unscientific stuff such as "reputation" as your only tool.

whatshisface 7y ago
I think you might want to qualify that by specific fields. For example if the paper you're reading is math or theoretical physics, then you actually do have all of the same capabilities as the author to verify it. As for publication bias, some studies are pre-registered. >Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. It is typical for a scientist to read hundreds and hundreds of papers every year, or sometimes even every month. Are you suggesting that they reproduce every one of them? You may be dismayed to find out that typically, they would reproduce none of them, and when the system is working its best they might reproduce one of them. Under usual circumstances, you and they have exactly the same replication ability: the ability to search the literature for a replication.

whatshisface 7y ago

I think you might want to qualify that by specific fields. For example if the paper you're reading is math or theoretical physics, then you actually do have all of the same capabilities as the author to verify it. As for publication bias, some studies are pre-registered. >Scientists who are in a position where they have the resources to _reproduce_ the articles should and do read them. It is typical for a scientist to read hundreds and hundreds of papers every year, or sometimes even every month. Are you suggesting that they reproduce every one of them? You may be dismayed to find out that typically, they would reproduce none of them, and when the system is working its best they might reproduce one of them. Under usual circumstances, you and they have exactly the same replication ability: the ability to search the literature for a replication.