4 ms·

How scientists fool themselves and how they can stop

96 points by DavidSJ 11y ago

timtas 11y ago
I think it's official, we moderns owe an apology to medieval barbers and conjurers.
- venomsnake 11y ago
  We don't - first we have correcting mechanisms in place. Second - the self deception ability is greater the softer/fuzzier the field is.
  - stewbrew 11y ago
    Who measures softness/fuzziness of a field? Who makes sure that measurement isn't biased? Just because you use maths doesn't imply your conclusions and theories are as precise and clean cut as 2+2.
    - gsteinb88 11y ago
      I don't like the use of soft/fuzzy for certain fields, but I do know what it means in this context: it's shorthand for "fields so complicated that we aren't even sure if the questions we're asking mean anything." Take physics vs. psychology as an example. The top story on HN right now is about standardizing the kilogram by answering two questions: how many silicon-28 atoms do you need to equal the mass of the current reference kilogram, and at what pount a watt balance can support the 1 kg weight. This depends on a bunch of terminology and assumptions, but we're pretty clear what those are. In psychology on the other hand (a field I have a huge amount of respect for, incidentally), we're trying to ask questions about human behavior; everything we ask about is taken within a cultural context, predicated on assumed levels of "normal" behavior, and even assumes that humans /are/ repeatable in some sense (and a lot of psychology studies may not be repeatable at all...) That repeatability question is a big one, I think: we're a lot more sure about our ability to prepare equivalent circumstances with silicon-28 atoms for an experiment than with people.
    - nmrm2 11y ago
      > Who measures softness/fuzziness of a field? There's an absolutely unambiguous distinction between the "hard" and "soft" sciences Social scientists hold themselves to fundamentally different standards on things like repeatability and predictive power and methodolgy than other fields. And whenever that difference can be quantified (e.g. statistical results), "different" means "lower". It's just a fact, and it's one most sociologists will freely admit, for example. You simply can't get the same sorts of results on humans that you can get on e.g. chemical reactions. Unfortunately, soft/hard comes along with a value judgement that's not necessarily appropriate. But pretending like there isn't a clear difference between what are now referred to as the soft and hard sciences is disinegenious.
      - arcanus 11y ago
        Generally agree with the sentiment, but I think the line is fuzzy along the periphery. For instance, economics has, I would claim, far weaker levels of model validation than engineering fields, but often is presented as a hard or 'close-to-hard' science.
texthompson 11y ago
It's too bad that there's not a more robust discussion about the institutional incentives that drive scientists to fool themselves.
- danieltillett 11y ago
  Exactly. It is very easy to fool yourself when your entire career is on the line if you don't.
- INTPnerd 11y ago
  Some things in science have become "immoral" to question. In a big way evolution, including macro evolution, has become the religion of science. If you even entertain the idea that macro evolution has not been happening, you might be labeled an evil heretic. You might be labeled anti-science. You might be laughed out of your field. You could get fired from your job. You might lose all your funding. All of this could happen even if you go about your research, testing, and analysis in a very objective, scientific way.
  - minority-one 11y ago
    Another sacred artifact in Science is WWII gas chambers. Fred Leuchter (manufacturer of equipment for gas executions), Nicholas Kollerstrom (Science Historian) and Germar Rudolf (Chemist) amongst others, were all outcast and even imprisoned (!) for daring to bring Science into this much politicized subject. No one benefits from censorship. Those listed above should have had their published findings refuted, not censored, if they are wrong. And they should definitely not have been thrown in jail for thought-crimes. Rudolf's published paper was actually peer-reviewed before he lost his PhD research position. This is absolutely outrageous and no one here has the courage to talk about it. It is a travesty and a mockery of what Science should be about.
- yiyus 11y ago
  This is a key aspect. There is a pressure (both institutional and self-imposed) to produce something original and unique, so trying to reproduce other people results is seen as a lost of time. Either you confirm previous results, which is nice but not something you can publish or write a thesis about; or you find they are wrong and then you have to tell the authors, which is something they may appreciate or not, and be exhaustive to prove it, which is something in which you may be interested or not. What I would like to see is that PhD students were required to reproduce some previous results as part of their thesis. This would increase the quality of scientific results and would prepare the candidates to do their own research (at difference of now, that they are put on front of a laboratory to get "interesting results" and they don't know what to do with it).
gaius 11y ago
A scientist's first instinct is "get more funding". This cannot help but bias their conclusions.
- andyjohnson0 11y ago
  Unnecessarily cynical. Its a problem that science funding makes scientist's careers so precarious, but nobody goes into science for the money.
  - cageface 11y ago
    Maybe nobody goes into science for the money, but if you don't focus on the money you won't stay in science very long.
  - dagw 11y ago
    "Get more funding" isn't the difference between making a little money vs. making a lot of money. It's the difference between making a little money and not having a job.
    - coherentpony 11y ago
      Depends on the scientist's position. Tenure-track and tenured faculty in the US typically have 9 months of their salary covered by the institution employing them. Only the remaining three months need be covered by grants (or teaching).
      - scott_s 11y ago
        But going from tenure track to tenure in most research universities does require getting grants.
- dalke 11y ago
  Do you expect to see any differences in the quality of science of those few scientist who are self-funded? (I heard a lecture once from a researcher who was on Wall Street, made millions, then switched to femtosecond lasers. He worked out of a university, and funded his own lab. So such people do exist.) If there are differences, are they measurable?
sjg007 11y ago
Until you have the technology to actually "see" something or measure it accurately, it is almost impossible to deduce reality from correlations.

96 points by DavidSJ 11y ago

timtas 11y ago

I think it's official, we moderns owe an apology to medieval barbers and conjurers.

venomsnake 11y ago
We don't - first we have correcting mechanisms in place. Second - the self deception ability is greater the softer/fuzzier the field is.
- stewbrew 11y ago
  Who measures softness/fuzziness of a field? Who makes sure that measurement isn't biased? Just because you use maths doesn't imply your conclusions and theories are as precise and clean cut as 2+2.
  - gsteinb88 11y ago
    I don't like the use of soft/fuzzy for certain fields, but I do know what it means in this context: it's shorthand for "fields so complicated that we aren't even sure if the questions we're asking mean anything." Take physics vs. psychology as an example. The top story on HN right now is about standardizing the kilogram by answering two questions: how many silicon-28 atoms do you need to equal the mass of the current reference kilogram, and at what pount a watt balance can support the 1 kg weight. This depends on a bunch of terminology and assumptions, but we're pretty clear what those are. In psychology on the other hand (a field I have a huge amount of respect for, incidentally), we're trying to ask questions about human behavior; everything we ask about is taken within a cultural context, predicated on assumed levels of "normal" behavior, and even assumes that humans /are/ repeatable in some sense (and a lot of psychology studies may not be repeatable at all...) That repeatability question is a big one, I think: we're a lot more sure about our ability to prepare equivalent circumstances with silicon-28 atoms for an experiment than with people.
  - nmrm2 11y ago
    > Who measures softness/fuzziness of a field? There's an absolutely unambiguous distinction between the "hard" and "soft" sciences Social scientists hold themselves to fundamentally different standards on things like repeatability and predictive power and methodolgy than other fields. And whenever that difference can be quantified (e.g. statistical results), "different" means "lower". It's just a fact, and it's one most sociologists will freely admit, for example. You simply can't get the same sorts of results on humans that you can get on e.g. chemical reactions. Unfortunately, soft/hard comes along with a value judgement that's not necessarily appropriate. But pretending like there isn't a clear difference between what are now referred to as the soft and hard sciences is disinegenious.
    - arcanus 11y ago
      Generally agree with the sentiment, but I think the line is fuzzy along the periphery. For instance, economics has, I would claim, far weaker levels of model validation than engineering fields, but often is presented as a hard or 'close-to-hard' science.

texthompson 11y ago

It's too bad that there's not a more robust discussion about the institutional incentives that drive scientists to fool themselves.

danieltillett 11y ago
Exactly. It is very easy to fool yourself when your entire career is on the line if you don't.
INTPnerd 11y ago
Some things in science have become "immoral" to question. In a big way evolution, including macro evolution, has become the religion of science. If you even entertain the idea that macro evolution has not been happening, you might be labeled an evil heretic. You might be labeled anti-science. You might be laughed out of your field. You could get fired from your job. You might lose all your funding. All of this could happen even if you go about your research, testing, and analysis in a very objective, scientific way.
- minority-one 11y ago
  Another sacred artifact in Science is WWII gas chambers. Fred Leuchter (manufacturer of equipment for gas executions), Nicholas Kollerstrom (Science Historian) and Germar Rudolf (Chemist) amongst others, were all outcast and even imprisoned (!) for daring to bring Science into this much politicized subject. No one benefits from censorship. Those listed above should have had their published findings refuted, not censored, if they are wrong. And they should definitely not have been thrown in jail for thought-crimes. Rudolf's published paper was actually peer-reviewed before he lost his PhD research position. This is absolutely outrageous and no one here has the courage to talk about it. It is a travesty and a mockery of what Science should be about.
yiyus 11y ago
This is a key aspect. There is a pressure (both institutional and self-imposed) to produce something original and unique, so trying to reproduce other people results is seen as a lost of time. Either you confirm previous results, which is nice but not something you can publish or write a thesis about; or you find they are wrong and then you have to tell the authors, which is something they may appreciate or not, and be exhaustive to prove it, which is something in which you may be interested or not. What I would like to see is that PhD students were required to reproduce some previous results as part of their thesis. This would increase the quality of scientific results and would prepare the candidates to do their own research (at difference of now, that they are put on front of a laboratory to get "interesting results" and they don't know what to do with it).

gaius 11y ago

A scientist's first instinct is "get more funding". This cannot help but bias their conclusions.

andyjohnson0 11y ago
Unnecessarily cynical. Its a problem that science funding makes scientist's careers so precarious, but nobody goes into science for the money.
- cageface 11y ago
  Maybe nobody goes into science for the money, but if you don't focus on the money you won't stay in science very long.
- dagw 11y ago
  "Get more funding" isn't the difference between making a little money vs. making a lot of money. It's the difference between making a little money and not having a job.
  - coherentpony 11y ago
    Depends on the scientist's position. Tenure-track and tenured faculty in the US typically have 9 months of their salary covered by the institution employing them. Only the remaining three months need be covered by grants (or teaching).
    - scott_s 11y ago
      But going from tenure track to tenure in most research universities does require getting grants.
dalke 11y ago
Do you expect to see any differences in the quality of science of those few scientist who are self-funded? (I heard a lecture once from a researcher who was on Wall Street, made millions, then switched to femtosecond lasers. He worked out of a university, and funded his own lab. So such people do exist.) If there are differences, are they measurable?

sjg007 11y ago

Until you have the technology to actually "see" something or measure it accurately, it is almost impossible to deduce reality from correlations.