4 ms·

Didn't you read the Memo? Science is a lot like sausage processing. The end result may range from the quite passable to the quite good, but if you actually saw

Didn't you read the Memo? Science is a lot like sausage processing. The end result may range from the quite passable to the quite good, but if you actually saw the raw materials and the process, you'd be more discriminant on what you put in your plate.

eudox 11y ago
In fairness to science, this is just psychology.
- mattkrause 11y ago
  No, really, it's not. There are very similar problems in most of the social AND natural sciences. In 2012, Amgen reported that they could only replicate 6/53 "landmark" cancer studies. Bayer had a similarly hard time replicating papers suggesting new drug targets in a smattering of fields including cardiology and oncology (14/67 successfully replicated).
  - mucker 11y ago
    Absolutely Matt. It is also shocking that most people believe that "peer reviewed" is synonymous with "reproduced study". It is not. One is editorial the other requires longer and heavier work.
  - Lawtonfogle 11y ago
    You are both right. The softer the science, the worse the effect. I trust physics more than biology, and I trust biology more than sociology. Well, that implies I trust sociology... I really don't. Anything softer than physiological psychology is basically in the 'don't trust this until you have verify it, and then still mostly don't trust it'.
- nsajko 11y ago
  You can't go insult a science based on the faults of a theory or a field.
drzaiusapelord 11y ago
Well, the problem is that the end result was proof of ego depletion. We can move the goalposts and say, "Wait, no this new nullification is the true end result," but those 100 studies ruled the roost for quite some time. The reality is that the end results are usually terrible and a lot of science out there is straight up junk science designed to chase grants and to lean on journal biases to get published, especially in the social sciences. The problem is we don't really have a better alternative so working with a clock that's right twice a day is much better than having no clocks at all. We might have hit organizational limits on how well we can do science on this level. With funding comes politics, with journals comes bias, with publishing comes perverse incentives, challenging big egos is difficult, unknowns have to 'pay their dues', small universities and institutions don't have influence compared to their larger peers, the sometimes political tenure process, etc. Hell, the article itself even calls out Carter's relatively young age or for not being tenured in an almost "Can we trust this young nobody," way.
- crpatino 11y ago
  > "Can we trust this young nobody," way. That's normal science at work. Old sages are biased, like everybody else. Fortunately, they are mortal too [1], and science keeps moving forward 'one funeral at a time', as the saying goes. The problem is with people who want to substitute faith in the Divine Providence of God, with faith in the (eventual) Unfailibility of Science. We are just a bunch of social primates, after all. We were never meant to peek into the Deep Mysteries of Existence, anyways... which does not mean we cannot use our relatively advanced intelects to figure out a bunch of clever and useful tricks. ======= edit: comment [1] [1] Something that is nightmare fuel to me is that one day we are going to stumble upon some serious life extending technology, and then we will all end up under the thumb of a bunch of ancient, obstinated fools who cannot/won't accept the limits of their own leadership skills.
  - Retra 11y ago
    Here's how I console myself: if those old obstinate fools are actually poor leaders, they will make worse decisions as a result, and thus not be as effective at marginalizing error as they should be, so they will be unable to avoid the unforeseen things that will get them killed.
entee 11y ago
One possible advantage the "harder" sciences have is that another study that builds on the previous one can in some ways act as a replication. If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y. Otherwise something is fishy, and often the second set of researchers will go back and reexamine the original result. It's rare that work in science is directly, point for point replicated, it's unsexy, expensive work. The fact that the harder sciences have somewhat of a built in control mitigates but by no means eliminates the risk of bad science surviving. I can't think of an analogous process in the softer sciences. Seeing if a phenomenon is replicated in a different context does not implicitly re-test the original work in the same way.
- asuffield 11y ago
  One of the biggest things about the "harder" sciences is that there's usually a whole engineering discipline behind them as well. Nothing reproduces a result as effectively as a host of engineers building real things on the back of it. If it doesn't work, we'll find out pretty soon, and if it does work then we'll have lots of evidence to show it. The problem with results like "ego depletion" is that their utility is fuzzy at best. With no clear application, we don't have a body of practical results using it.
- jpfed 11y ago
  >If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y. There is no guarantee (and it often isn't the case) that the phenotype observed in the first organism is purely the result of gene X; its effects may be context-dependent. To take this a little further, I think it's plausible that a soft scientific question differs from a hard one specifically in terms of how easy it is to separate the phenomenon in question from its context.
havefunwiththat 11y ago
Is "discriminant" a word now? I thought performant had pretty much exhausted the possibilities.

eudox 11y ago

In fairness to science, this is just psychology.

mattkrause 11y ago
No, really, it's not. There are very similar problems in most of the social AND natural sciences. In 2012, Amgen reported that they could only replicate 6/53 "landmark" cancer studies. Bayer had a similarly hard time replicating papers suggesting new drug targets in a smattering of fields including cardiology and oncology (14/67 successfully replicated).
- mucker 11y ago
  Absolutely Matt. It is also shocking that most people believe that "peer reviewed" is synonymous with "reproduced study". It is not. One is editorial the other requires longer and heavier work.
- Lawtonfogle 11y ago
  You are both right. The softer the science, the worse the effect. I trust physics more than biology, and I trust biology more than sociology. Well, that implies I trust sociology... I really don't. Anything softer than physiological psychology is basically in the 'don't trust this until you have verify it, and then still mostly don't trust it'.
nsajko 11y ago
You can't go insult a science based on the faults of a theory or a field.

drzaiusapelord 11y ago

Well, the problem is that the end result was proof of ego depletion. We can move the goalposts and say, "Wait, no this new nullification is the true end result," but those 100 studies ruled the roost for quite some time. The reality is that the end results are usually terrible and a lot of science out there is straight up junk science designed to chase grants and to lean on journal biases to get published, especially in the social sciences. The problem is we don't really have a better alternative so working with a clock that's right twice a day is much better than having no clocks at all. We might have hit organizational limits on how well we can do science on this level. With funding comes politics, with journals comes bias, with publishing comes perverse incentives, challenging big egos is difficult, unknowns have to 'pay their dues', small universities and institutions don't have influence compared to their larger peers, the sometimes political tenure process, etc. Hell, the article itself even calls out Carter's relatively young age or for not being tenured in an almost "Can we trust this young nobody," way.

crpatino 11y ago
> "Can we trust this young nobody," way. That's normal science at work. Old sages are biased, like everybody else. Fortunately, they are mortal too [1], and science keeps moving forward 'one funeral at a time', as the saying goes. The problem is with people who want to substitute faith in the Divine Providence of God, with faith in the (eventual) Unfailibility of Science. We are just a bunch of social primates, after all. We were never meant to peek into the Deep Mysteries of Existence, anyways... which does not mean we cannot use our relatively advanced intelects to figure out a bunch of clever and useful tricks. ======= edit: comment [1] [1] Something that is nightmare fuel to me is that one day we are going to stumble upon some serious life extending technology, and then we will all end up under the thumb of a bunch of ancient, obstinated fools who cannot/won't accept the limits of their own leadership skills.
- Retra 11y ago
  Here's how I console myself: if those old obstinate fools are actually poor leaders, they will make worse decisions as a result, and thus not be as effective at marginalizing error as they should be, so they will be unable to avoid the unforeseen things that will get them killed.

entee 11y ago

One possible advantage the "harder" sciences have is that another study that builds on the previous one can in some ways act as a replication. If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y. Otherwise something is fishy, and often the second set of researchers will go back and reexamine the original result. It's rare that work in science is directly, point for point replicated, it's unsexy, expensive work. The fact that the harder sciences have somewhat of a built in control mitigates but by no means eliminates the risk of bad science surviving. I can't think of an analogous process in the softer sciences. Seeing if a phenomenon is replicated in a different context does not implicitly re-test the original work in the same way.

asuffield 11y ago
One of the biggest things about the "harder" sciences is that there's usually a whole engineering discipline behind them as well. Nothing reproduces a result as effectively as a host of engineers building real things on the back of it. If it doesn't work, we'll find out pretty soon, and if it does work then we'll have lots of evidence to show it. The problem with results like "ego depletion" is that their utility is fuzzy at best. With no clear application, we don't have a body of practical results using it.
jpfed 11y ago
>If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y. There is no guarantee (and it often isn't the case) that the phenotype observed in the first organism is purely the result of gene X; its effects may be context-dependent. To take this a little further, I think it's plausible that a soft scientific question differs from a hard one specifically in terms of how easy it is to separate the phenomenon in question from its context.

havefunwiththat 11y ago

Is "discriminant" a word now? I thought performant had pretty much exhausted the possibilities.