15 ms·

> And before someone asks "But what if it's .. instead?": Yes, they controlled for lots of things On Reddit's /r/science board, there's a base rule that "criti

> And before someone asks "But what if it's .. instead?": Yes, they controlled for lots of things On Reddit's /r/science board, there's a base rule that "criticism of published work should assume basic competence of the researchers and reviewers". I think it would benefit HN if we all made the same assumption.

ekianjo 4y ago
> On Reddit's /r/science board, there's a base rule that "criticism of published work should assume basic competence of the researchers and reviewers". Since most studies/publications cant be replicated, I'd say basic competence is not warranted.
vorpalhex 4y ago
As someone who reads a lot of published research, no, that is a terrible rule and a lot of garbage gets published. Both historically and now there is no shortage of bad science that gets published, or good science that gets really misreported.
- falcolas 4y ago
  If you don't trust studies at such a fundamental level, what is the value in reading and commenting on such threads?
  - tuatoru 4y ago
    I don't know what value vorpalhex gets from doing so, but I got value from the gp comment.
  - vorpalhex 4y ago
    Studies don't have value just because they are a study. Put another way, just because Cosmo publishes an article doesn't mean it's true - but Cosmo does also publish true articles. Plenty of studies are great and move our understanding forward. More studies answer a very narrow - but important - question. Some studies are just inconclusive, but have value too. You should read studies. You should evaluate the quality of studies as part of integrating them and their conclusions. At the same time just because something was "a study" or got published doesn't make it true or good or useful, anymore than being written in blue ink makes something truthful. The value of science is that it does not require faith.
  - JoshuaDavid 4y ago
    Not OP but If someone were advocating making policy decisions based on astrological charts, I would read and comment in such threads despite not trusting astrology on a fundamental level. I personally would go quite as far as to say that purely observational studies have as much predictive value as astrology, but "observational studies are as reliable as astrology" is a better first-order approximation of my opinions than "observational studies are as reliable as large-n randomized controlled trials".
- eszaq 4y ago
  A specific form of bad research which could be at play here: The researchers programmatically searched the space of possible control variables to include until they came up with a model which maximized the apparent effect size for coffee, so they could publish an interesting & widely cited paper that looked good on their resumes. With N different control variables to either include or ignore, that's 2^N possible sets of control variables. Odds are decent at least one of those regressions has a large effect size for coffee. I would trust this sort of research more if instead of publishing a particular set of control variables obtained by an unspecified method, the researchers chose 100 of those 2^N possible sets of control variables at random, then published the average effect size from the 100 resulting regressions. Ideally they would make the code to reproduce this average effect size publicly available, so anyone could easily replicate using another 100 randomly generated regressions.
FastMonkey 4y ago
I don't think it's questioning anyone's competence to suggest that a single scientific result might have confounding factors. This is one study done by one team. If the result was repeated in several studies, all coming at it with different viewpoints then I'd start to believe the result. In this particular case though, there seems to be new research coming out every few months saying coffee is good or bad for you. If the results look like that then it's hard to put much weight on any single one of them.
- falcolas 4y ago
  > I don't think it's questioning anyone's competence to suggest that a single scientific result might have confounding factors Honestly - I do. Basic competence in creating a study would definitely include identifying confounding factors that are directly relevant to the study being conducted. It's a basic part of the scientific method - both in creating the hypothesis and creating a test against that hypothesis.
dekhn 4y ago
This is a fundamentally wrong rule because, as we have learned, researchers and reviewers basic competence has come into play repeatedly. The further you get from quantitative physics, the worse it gets. That said, if you are going to criticize something based on the idea "this result doesn't agree with my preconcieved notions so I'm going to say that their sample size was too small, the effect size was too small, or they made some sort of incorrect calculation, and conclude the paper can't be trusted", you owe it to yourself to read the paper carefully enough that you can make the determination they made a major error. Unlike most people, I read the conclusion, then the methods section. For most papers I can't get enough from the methods section to conclude that I trust the authors to make their claim, because most methods sections are missing major details on how the corrections/controls were made.
- falcolas 4y ago
  Should we also look at final code assuming the programmer and their code reviewers lack basic competence? Or do we assume they made a best effort given their abilities and may have still made mistakes? I believe the latter, and the rule I mentioned also leads to that assumption. Otherwise, why read the studies at all? But that's not what people here do. > missing major details on how the corrections/controls were made I also make an assumption that some corrections/controls are so obvious within their field of study that they aren't worth documenting - that it's only "average joe" readers who assume they aren't being done. It is, in a way, akin to asking why programmers didn't document how their JWT is secured.
  - dekhn 4y ago
    I don't compare software engineers and scientists by the same metrics. The important thing about a code review request is that it contains everything required for me to reproduce the effects of the code change in my own hands. Science does not require that- instead, the methods is typically the "least descriptive and helpful representation of what was done" (especially in the case of very complex biological experiments) and can't be reproduced with nontrivial effort. BTW I've definitely gone into code reviews assuming the programmer lacks basic competence and been right. Other times, I've had to rollback other people's code (or stop a rollout) because an ostensibly genius programmer who made a change (and got a review from a starry-eyed junior) wasn't using their basic competence or testing their change at all.
Jtype 4y ago
I don't think reddit and in particular /r/science should be held aloft as a beacon of reason and good thinking. The pseudo science and junk that is regularly upvoted there proves that they should more frequently question the basic competence of the researchers and reviewers.

ekianjo 4y ago

> On Reddit's /r/science board, there's a base rule that "criticism of published work should assume basic competence of the researchers and reviewers". Since most studies/publications cant be replicated, I'd say basic competence is not warranted.

vorpalhex 4y ago

As someone who reads a lot of published research, no, that is a terrible rule and a lot of garbage gets published. Both historically and now there is no shortage of bad science that gets published, or good science that gets really misreported.

falcolas 4y ago
If you don't trust studies at such a fundamental level, what is the value in reading and commenting on such threads?
- tuatoru 4y ago
  I don't know what value vorpalhex gets from doing so, but I got value from the gp comment.
- vorpalhex 4y ago
  Studies don't have value just because they are a study. Put another way, just because Cosmo publishes an article doesn't mean it's true - but Cosmo does also publish true articles. Plenty of studies are great and move our understanding forward. More studies answer a very narrow - but important - question. Some studies are just inconclusive, but have value too. You should read studies. You should evaluate the quality of studies as part of integrating them and their conclusions. At the same time just because something was "a study" or got published doesn't make it true or good or useful, anymore than being written in blue ink makes something truthful. The value of science is that it does not require faith.
- JoshuaDavid 4y ago
  Not OP but If someone were advocating making policy decisions based on astrological charts, I would read and comment in such threads despite not trusting astrology on a fundamental level. I personally would go quite as far as to say that purely observational studies have as much predictive value as astrology, but "observational studies are as reliable as astrology" is a better first-order approximation of my opinions than "observational studies are as reliable as large-n randomized controlled trials".
eszaq 4y ago
A specific form of bad research which could be at play here: The researchers programmatically searched the space of possible control variables to include until they came up with a model which maximized the apparent effect size for coffee, so they could publish an interesting & widely cited paper that looked good on their resumes. With N different control variables to either include or ignore, that's 2^N possible sets of control variables. Odds are decent at least one of those regressions has a large effect size for coffee. I would trust this sort of research more if instead of publishing a particular set of control variables obtained by an unspecified method, the researchers chose 100 of those 2^N possible sets of control variables at random, then published the average effect size from the 100 resulting regressions. Ideally they would make the code to reproduce this average effect size publicly available, so anyone could easily replicate using another 100 randomly generated regressions.

FastMonkey 4y ago

I don't think it's questioning anyone's competence to suggest that a single scientific result might have confounding factors. This is one study done by one team. If the result was repeated in several studies, all coming at it with different viewpoints then I'd start to believe the result. In this particular case though, there seems to be new research coming out every few months saying coffee is good or bad for you. If the results look like that then it's hard to put much weight on any single one of them.

falcolas 4y ago
> I don't think it's questioning anyone's competence to suggest that a single scientific result might have confounding factors Honestly - I do. Basic competence in creating a study would definitely include identifying confounding factors that are directly relevant to the study being conducted. It's a basic part of the scientific method - both in creating the hypothesis and creating a test against that hypothesis.

dekhn 4y ago

This is a fundamentally wrong rule because, as we have learned, researchers and reviewers basic competence has come into play repeatedly. The further you get from quantitative physics, the worse it gets. That said, if you are going to criticize something based on the idea "this result doesn't agree with my preconcieved notions so I'm going to say that their sample size was too small, the effect size was too small, or they made some sort of incorrect calculation, and conclude the paper can't be trusted", you owe it to yourself to read the paper carefully enough that you can make the determination they made a major error. Unlike most people, I read the conclusion, then the methods section. For most papers I can't get enough from the methods section to conclude that I trust the authors to make their claim, because most methods sections are missing major details on how the corrections/controls were made.

falcolas 4y ago
Should we also look at final code assuming the programmer and their code reviewers lack basic competence? Or do we assume they made a best effort given their abilities and may have still made mistakes? I believe the latter, and the rule I mentioned also leads to that assumption. Otherwise, why read the studies at all? But that's not what people here do. > missing major details on how the corrections/controls were made I also make an assumption that some corrections/controls are so obvious within their field of study that they aren't worth documenting - that it's only "average joe" readers who assume they aren't being done. It is, in a way, akin to asking why programmers didn't document how their JWT is secured.
- dekhn 4y ago
  I don't compare software engineers and scientists by the same metrics. The important thing about a code review request is that it contains everything required for me to reproduce the effects of the code change in my own hands. Science does not require that- instead, the methods is typically the "least descriptive and helpful representation of what was done" (especially in the case of very complex biological experiments) and can't be reproduced with nontrivial effort. BTW I've definitely gone into code reviews assuming the programmer lacks basic competence and been right. Other times, I've had to rollback other people's code (or stop a rollout) because an ostensibly genius programmer who made a change (and got a review from a starry-eyed junior) wasn't using their basic competence or testing their change at all.

Jtype 4y ago

I don't think reddit and in particular /r/science should be held aloft as a beacon of reason and good thinking. The pseudo science and junk that is regularly upvoted there proves that they should more frequently question the basic competence of the researchers and reviewers.