4 ms·

I was thinking of making a database of only studies that we can “trust” What does HN think? Had a tough time determining the barometer for what is trustable an

I was thinking of making a database of only studies that we can “trust” What does HN think? Had a tough time determining the barometer for what is trustable and what isn’t

nathan_compton 3y ago
There is no simple way to do this. If it were easy to identify trustworthy research we wouldn't be having this discussion. From a slightly broader point of view, I'd argue that the idea that things can be easily or even meaningfully sorted into trustworthy and non-trustworthy is epistemologically facile, at the very least.
ajhurliman 3y ago
Isn’t that effectively just a publishing company?
some_random 3y ago
I'd love that, but determining trust is going to be very hard. For example, where would you put a study that is 100% factually accurate, but yields intentionally misleading statistics for political reasons?
zachwill 3y ago
Examine.com does something similar for health related studies, and I've found it valuable in the past.
ARandumGuy 3y ago
Unfortunately there's a wide range between "obviously trustworthy" and "obviously bullshit". Many bad studies require a lot of domain knowledge to understand why they're bad. Some controversial studies were widely discredited by the rest of the field, but actually turned out to be correct. Many studies are worth doing, but shouldn't be used to draw any meaningful conclusions (such as pilot studies). And finally, there are some studies that seem legitimate, but are actually just lying about their data. And that's before you get into ideological debates and political problems. We like to think scientists are bastions of rationality, but they're just as susceptible to petty infighting as any other group of people. A database of trustworthy studies would be an incredibly valuable tool. Unfortunately, it's a tool that is nigh impossible to actually create.
- anonymouskimmer 3y ago
  > Many bad studies require a lot of domain knowledge to understand why they're bad. I went to grad school after some years practicing in the discipline. During those earlier years I neglected to introduce a Kozak sequence the first time constructing a protein expression cassette. These sequences, which are on the 5 prime region immediately adjacent to the protein open reading frame, are almost always required for decent translation of mRNA to protein in eukaryotes. And minor modifications to the Kozak sequence can have varying impact on protein expression. Since then this has always been top of my mind when doing such design. So in grad school in a class we had various papers assigned to read and answer questions on. One peer-reviewed paper split a 5 prime untranslated region (an mRNA regulatory element) into two parts to determine which parts were most necessary for regulating expression of the protein. They placed these two parts directly adjacent to the protein open reading frame. And unfortunately, for the first half of the untranslated region, they neglected to keep the native Kozak sequence. Not unexpectedly, there was almost no protein expression from this. They concluded that the first half of the untranslated region was both necessary and sufficient for down regulation of expression. I pointed out their fatal flaw and seriously impressed my professor who had only done a cursory reading of the paper before assigning it. I don't understand how this passed peer-review, but it did. I guess the reviewers didn't look at the mRNA sequence in detail. If they had, they would have noticed the altered Kozak site and would have required a further experiment with the native Kozak sequence. The study was actually fine in basic design, the author just made an easy mistake, and no one caught it.
naasking 3y ago
You can trust some claims from a study but not other claims. Maybe a database of trustworthy claims with the studies and commentary on why they support the claim. With error bars!
- l33t233372 3y ago
  HN reinvents the “text book.”
  - naasking 3y ago
    I don't know, I've read some pretty terrible textbooks.

I was thinking of making a database of only studies that we can “trust” What does HN think? Had a tough time determining the barometer for what is trustable and what isn’t

nathan_compton 3y ago

There is no simple way to do this. If it were easy to identify trustworthy research we wouldn't be having this discussion. From a slightly broader point of view, I'd argue that the idea that things can be easily or even meaningfully sorted into trustworthy and non-trustworthy is epistemologically facile, at the very least.

ajhurliman 3y ago

Isn’t that effectively just a publishing company?

some_random 3y ago

I'd love that, but determining trust is going to be very hard. For example, where would you put a study that is 100% factually accurate, but yields intentionally misleading statistics for political reasons?

zachwill 3y ago

Examine.com does something similar for health related studies, and I've found it valuable in the past.

ARandumGuy 3y ago

Unfortunately there's a wide range between "obviously trustworthy" and "obviously bullshit". Many bad studies require a lot of domain knowledge to understand why they're bad. Some controversial studies were widely discredited by the rest of the field, but actually turned out to be correct. Many studies are worth doing, but shouldn't be used to draw any meaningful conclusions (such as pilot studies). And finally, there are some studies that seem legitimate, but are actually just lying about their data. And that's before you get into ideological debates and political problems. We like to think scientists are bastions of rationality, but they're just as susceptible to petty infighting as any other group of people. A database of trustworthy studies would be an incredibly valuable tool. Unfortunately, it's a tool that is nigh impossible to actually create.

anonymouskimmer 3y ago
> Many bad studies require a lot of domain knowledge to understand why they're bad. I went to grad school after some years practicing in the discipline. During those earlier years I neglected to introduce a Kozak sequence the first time constructing a protein expression cassette. These sequences, which are on the 5 prime region immediately adjacent to the protein open reading frame, are almost always required for decent translation of mRNA to protein in eukaryotes. And minor modifications to the Kozak sequence can have varying impact on protein expression. Since then this has always been top of my mind when doing such design. So in grad school in a class we had various papers assigned to read and answer questions on. One peer-reviewed paper split a 5 prime untranslated region (an mRNA regulatory element) into two parts to determine which parts were most necessary for regulating expression of the protein. They placed these two parts directly adjacent to the protein open reading frame. And unfortunately, for the first half of the untranslated region, they neglected to keep the native Kozak sequence. Not unexpectedly, there was almost no protein expression from this. They concluded that the first half of the untranslated region was both necessary and sufficient for down regulation of expression. I pointed out their fatal flaw and seriously impressed my professor who had only done a cursory reading of the paper before assigning it. I don't understand how this passed peer-review, but it did. I guess the reviewers didn't look at the mRNA sequence in detail. If they had, they would have noticed the altered Kozak site and would have required a further experiment with the native Kozak sequence. The study was actually fine in basic design, the author just made an easy mistake, and no one caught it.

naasking 3y ago

You can trust some claims from a study but not other claims. Maybe a database of trustworthy claims with the studies and commentary on why they support the claim. With error bars!

l33t233372 3y ago
HN reinvents the “text book.”
- naasking 3y ago
  I don't know, I've read some pretty terrible textbooks.