7 ms·

Cargo Cult Science (1974)

100ideas 10y ago
16th time this has been posted to HN https://hn.algolia.com/?query=cargo%20cult%20science https://hn.algolia.com/?query=cargo%20cult%20science
- zyxley 10y ago
  "Today's lucky 10,000", etc.
  - schoen 10y ago
    (from https://www.xkcd.com/1053/ https://www.xkcd.com/1053/, for the lucky 10,000 readers who haven't seen this comic before!)
- dang 10y ago
  It's been three years since there was a significant discussion, though, so we won't treat it as a dupe. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
- harry8 10y ago
  I wouldn't complain if it were posted 160 times. It's brilliant, amazing, relevant stuff that hasn't dated. I've re-read it at least 16 times since first encountering it in "Surely you're joking Mr Feynman" in the 80s.
  - m1n1 10y ago
    it's been so long, why are there typos in it?!
pvsukale1 10y ago
today is richard feynman's bday! happy bday feynman!!
deleted 10y ago
[deleted]
NoMoreNicksLeft 10y ago
I wonder how many cheer this, then turn around and do the sorts of things he was complaining about.
jhallenworld 10y ago
Well we should talk about Cargo Cult Management. I suspect most management is Cargo Cult :-) https://gigaom.com/2009/06/21/cargo-cult-management/ https://gigaom.com/2009/06/21/cargo-cult-management/
- wycx 10y ago
  Very much so. My pet theory is that most business management practices evolved from manufacturing production line environments, and the application of these management practices to, say, running a university or research facility is cargo cult in the extreme.
btilly 10y ago
Everyone who cares about science has known his point for over 40 years now. But Psychology, the subject he was discussing, continued to not take figuring out how to replicate seriously until the recent replication crisis demonstrated exactly HOW bad it was in terms that they couldn't ignore. See https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis for more on that. And now it is a hot topic with lots of, "Look at what we've figured out about how to make our research better!" With very little discussion of the fact that this was something they should have known all along.
- zevkirsh 10y ago
  yes the problem is politics , or what you might call institutionalized group psychology of those with access to spending tax dollars, and the legislators they need to 'persuade' to get the money spent. science is not an art , it is generally an institutionally funded social behavior. the art of science is such that a lot of shitty ass sciece research with poorly constructed hypothesis and protocol gets funded............ can we do a better job? maybe. but the answer lies in better policy analysts and politicians, not in better 'scientists'. on heuristic for helping pinpoint bad and poorly constructed research is to look for where the most 'growth' in research has occured the quickest; and by that-----i mean look for the research 'bubbles'. why? as with art in general, and clothign and every sphere of human behavior-----that which gets popular very quickly is usually by nature fleeting and without the most substance, like fast growing weeds. that which grows slowly and withstands the test of time usually resembles an oak in quality. that does not mean that fast growth is always undesireable. you want fast growth in areas where there is great potential and low hanging fruit. it is good that bubbles occur SOMETIMES. BUT NOT ALWAYS. some bubbles are poorly advised. you must crack a few eggs to make an ommelet but sometimes you are just breaking eggs inside the carton without ever making breakfast. but regardless of whether or not a bubble is good or bad, all bubbles still must pop at some point to let the overgrowth and wasteful growth be cleared. all forests must eventually have a forest fire. the longer fire and destruction is suppressed beyond a certain point........the greater the amount of destruction is going to be when it finally comes. and in general, this is why our current economic bubble is so worrisome
joelthelion 10y ago
This applies extremely well to programming and software engineering in particular. "Best practices" are often a good excuse to avoid thinking hard about the problem at hand.
pjlegato 10y ago
The entire "agile" methodology strikes me as a prominent example of a software engineering cargo cult. Yes, good projects have been executed successfully using agile methodology. No, this does not mean that using agile caused these projects to work out well. Most agile projects fail, just like most non-agile projects. There is no evidence whatsoever that agile methodology causes projects to work out well -- but there is a whole mini-industry of training, books, seminars, "scrum masters" and so on built around the idea that it does.
- projectileboy 10y ago
  I agree, although I don't see that there's anything uniquely terrible about "agile". You could say the same thing about almost anything in our industry.
- solipsism 10y ago
  There's very little evidence whatsoever about any software development methodology or practice working well or poorly. That said, most experienced developers I've interacted with consider most of the Agile principles to be wise -- either common sense, or shown to be true via decades of experience. Here are the principles: * Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. * Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. * Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. * Business people and developers must work together daily throughout the project. * Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. * The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. * Working software is the primary measure of progress. * Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. * Continuous attention to technical excellence and good design enhances agility. * Simplicity--the art of maximizing the amount of work not done--is essential. * The best architectures, requirements, and designs emerge from self-organizing teams. * At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Which of these would you argue aren't wise principles for all of us to use when developing software? Now, if your beef is with Scrum (or some other branded methodology) in particular, I'd be more inclined to agree.
maxxxxx 10y ago
I think there is a lot of cargo culting going on everywhere. The lucky ones pick the right cult. How do you know whether your model of how the world works is right?
- mcnamaratw 10y ago
  Well, in EE, you know your model of your circuit is right if the measurements are all consistent with V=IR and so forth! The world, that's a little harder.
- matt_wulfeck 10y ago
  Exactly. If we agree there isn't one right way to do things, what's wrong with picking one and getting on with things? I see no benefit into falling onto the sword just to prove a theoretical point. Too much work to do.
- sinxoveretothex 10y ago
  You're looking at this the wrong way. The question is not "does reality match this model?" but rather "how well does this model match reality?". You never know exactly to what exact extent your map matches the territory. But you can have a pretty good idea. Like, we know (to the extent that we can "know" anything) that models predicting the Sun will rise tomorrow match reality better than those which don't. We know that models which predict that turning on a switch will cause the lightbulb to emit light are more accurate that those who say that it's magic or mysterious. To quote Feynman: > We’ve learned from experience that the truth will out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature’s phenomena will agree or they’ll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven’t tried to be very careful in this kind of work. And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in Cargo Cult Science.
  - maxxxxx 10y ago
    In physics or other hard sciences it's possible to validate a lot of models. I was more thinking about other areas like politics, management or software engineering issues. There it's often not possible to validate your model. It's even hard to tell if something goes well or not. And the feedback loop from decision to result can take years in many cases. So you pretty much are forced to do cargo culting.
kragen 10y ago
In CS (Systems, not Theory) research there's a big push now to move toward reproducibility by publishing code and dependencies; https://twitter.com/kragen/status/727220736633524228 https://twitter.com/kragen/status/727220736633524228 talks a bit about some of the historical context and details. Hackademia has a couple of synergistic efforts going on, directed mostly at securing the free-software distribution infrastructure and somewhat at simplifying system administration: on one hand you have reproducible builds being done by Tor, Mozilla, and Debian, and on the other hand you have Nix, Guix, and the like pushing purely-functional content-addressable package management. The upshot of all of this is that, in theory, in the near future, it will be practical to publish machine-executable instructions for reproducing the bit-identical executables and other output products you used in your research, at a cost of only a few gigabytes, so they can be exactly reproduced at any time in the future, and then modified to do new experiments; and it will be straightforward to make archival copies of this entire set of dependencies so that it won't be lost because some random guy died or decided to pull his modules from npm. Mike Hoye famously complained about how far we are from this in a Tweetstorm the other day: https://twitter.com/mhoye/status/725010388547330048 https://twitter.com/mhoye/status/725010388547330048 The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem. Nobody really knows what went wrong, but our best guess at this point is that the Sprite operating system had unknown bugs in it that gave a performance advantage to log-structured filesystems. (ChuckMcM may want to weigh in, since his company at the time (NetApp) was getting record-setting performance with WAFL, which is very similar to a LFS in some ways, but not others.) Jan Vitek, Jean Yang, Sam Tobin-Hochstadt, and Shriram Krishnamurthi are four prominent researchers who are advocating this change; they and others have created a new institution called "Artifact Evaluation Committees", which review the artifacts submitted by authors to see if they are able to reproduce the results the authors report. They've written an overview of it here, and they seem to already be getting significant results: http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-dont-publish-code.html http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-d... Hopefully over time this will create a bubble of real, reproducible science inside of the systems part of Computer "Science" research. [Earlier versions of this comment did not mention Krishnamurthi or Vitek and the synergistic work in hackademia, which was a combination of cluelessness and carelessness on my part. I regret the omissions.]
kirrent 10y ago
The parts that always strike me the most about this is that in 1974 Feynman lamented a lack of scientific rigour in deciding how we teach and how we manage crime and the criminal justice system. In both cases, despite 40 years having passed, it's hard to say there's been any large improvement in the application of rigour.
- abecedarius 10y ago
  There's some progress: https://en.wikipedia.org/wiki/Sequential_lineups https://en.wikipedia.org/wiki/Sequential_lineups The part that boggled me was how recent (and incomplete) was the introduction of blinding to police lineups. Apparently it really was (and is?) common for the witness to pick out a suspect, and for the cop to know the 'real' suspect and say things like "Are you sure? Take your time." From another Feynman talk: "This is not yet a scientific age."
- naivepiano 10y ago
  Do you have any link to that speech/article?
dschiptsov 10y ago
He probably could have been delighted to see all these "pendulums going in both directions at the same time" and other marvels of modern science we so often see on this site.
m1n1 10y ago
Ancient typos She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to __nut__ try to repeat psychological experiments, but only to change the conditions and see __hat__ happens.
Pica_soO 10y ago
The problem can not be solved by education alone. Education alone does not prevent the minds from making mistakes. Its about building and maintaining social structures that destroy cargo science and at the same time give freedom to groups/individuals who abstain from it. On envisions a credit system, in which you earn credibility by proofing something you did works- or by disproofing something that was wrong (obviously you get all credits that proof held). New credits enters the system with every entering scholar.

16th time this has been posted to HN https://hn.algolia.com/?query=cargo%20cult%20science https://hn.algolia.com/?query=cargo%20cult%20science

zyxley 10y ago
"Today's lucky 10,000", etc.
- schoen 10y ago
  (from https://www.xkcd.com/1053/ https://www.xkcd.com/1053/, for the lucky 10,000 readers who haven't seen this comic before!)
dang 10y ago
It's been three years since there was a significant discussion, though, so we won't treat it as a dupe. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
harry8 10y ago
I wouldn't complain if it were posted 160 times. It's brilliant, amazing, relevant stuff that hasn't dated. I've re-read it at least 16 times since first encountering it in "Surely you're joking Mr Feynman" in the 80s.
- m1n1 10y ago
  it's been so long, why are there typos in it?!

pvsukale1 10y ago

today is richard feynman's bday! happy bday feynman!!

deleted 10y ago

[deleted]

NoMoreNicksLeft 10y ago

I wonder how many cheer this, then turn around and do the sorts of things he was complaining about.

jhallenworld 10y ago

Well we should talk about Cargo Cult Management. I suspect most management is Cargo Cult :-) https://gigaom.com/2009/06/21/cargo-cult-management/ https://gigaom.com/2009/06/21/cargo-cult-management/

wycx 10y ago
Very much so. My pet theory is that most business management practices evolved from manufacturing production line environments, and the application of these management practices to, say, running a university or research facility is cargo cult in the extreme.

btilly 10y ago

Everyone who cares about science has known his point for over 40 years now. But Psychology, the subject he was discussing, continued to not take figuring out how to replicate seriously until the recent replication crisis demonstrated exactly HOW bad it was in terms that they couldn't ignore. See https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis for more on that. And now it is a hot topic with lots of, "Look at what we've figured out about how to make our research better!" With very little discussion of the fact that this was something they should have known all along.

zevkirsh 10y ago
yes the problem is politics , or what you might call institutionalized group psychology of those with access to spending tax dollars, and the legislators they need to 'persuade' to get the money spent. science is not an art , it is generally an institutionally funded social behavior. the art of science is such that a lot of shitty ass sciece research with poorly constructed hypothesis and protocol gets funded............ can we do a better job? maybe. but the answer lies in better policy analysts and politicians, not in better 'scientists'. on heuristic for helping pinpoint bad and poorly constructed research is to look for where the most 'growth' in research has occured the quickest; and by that-----i mean look for the research 'bubbles'. why? as with art in general, and clothign and every sphere of human behavior-----that which gets popular very quickly is usually by nature fleeting and without the most substance, like fast growing weeds. that which grows slowly and withstands the test of time usually resembles an oak in quality. that does not mean that fast growth is always undesireable. you want fast growth in areas where there is great potential and low hanging fruit. it is good that bubbles occur SOMETIMES. BUT NOT ALWAYS. some bubbles are poorly advised. you must crack a few eggs to make an ommelet but sometimes you are just breaking eggs inside the carton without ever making breakfast. but regardless of whether or not a bubble is good or bad, all bubbles still must pop at some point to let the overgrowth and wasteful growth be cleared. all forests must eventually have a forest fire. the longer fire and destruction is suppressed beyond a certain point........the greater the amount of destruction is going to be when it finally comes. and in general, this is why our current economic bubble is so worrisome

joelthelion 10y ago

This applies extremely well to programming and software engineering in particular. "Best practices" are often a good excuse to avoid thinking hard about the problem at hand.

pjlegato 10y ago

The entire "agile" methodology strikes me as a prominent example of a software engineering cargo cult. Yes, good projects have been executed successfully using agile methodology. No, this does not mean that using agile caused these projects to work out well. Most agile projects fail, just like most non-agile projects. There is no evidence whatsoever that agile methodology causes projects to work out well -- but there is a whole mini-industry of training, books, seminars, "scrum masters" and so on built around the idea that it does.

projectileboy 10y ago
I agree, although I don't see that there's anything uniquely terrible about "agile". You could say the same thing about almost anything in our industry.
solipsism 10y ago
There's very little evidence whatsoever about any software development methodology or practice working well or poorly. That said, most experienced developers I've interacted with consider most of the Agile principles to be wise -- either common sense, or shown to be true via decades of experience. Here are the principles: * Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. * Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. * Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. * Business people and developers must work together daily throughout the project. * Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. * The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. * Working software is the primary measure of progress. * Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. * Continuous attention to technical excellence and good design enhances agility. * Simplicity--the art of maximizing the amount of work not done--is essential. * The best architectures, requirements, and designs emerge from self-organizing teams. * At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Which of these would you argue aren't wise principles for all of us to use when developing software? Now, if your beef is with Scrum (or some other branded methodology) in particular, I'd be more inclined to agree.

maxxxxx 10y ago

I think there is a lot of cargo culting going on everywhere. The lucky ones pick the right cult. How do you know whether your model of how the world works is right?

mcnamaratw 10y ago
Well, in EE, you know your model of your circuit is right if the measurements are all consistent with V=IR and so forth! The world, that's a little harder.
matt_wulfeck 10y ago
Exactly. If we agree there isn't one right way to do things, what's wrong with picking one and getting on with things? I see no benefit into falling onto the sword just to prove a theoretical point. Too much work to do.
sinxoveretothex 10y ago
You're looking at this the wrong way. The question is not "does reality match this model?" but rather "how well does this model match reality?". You never know exactly to what exact extent your map matches the territory. But you can have a pretty good idea. Like, we know (to the extent that we can "know" anything) that models predicting the Sun will rise tomorrow match reality better than those which don't. We know that models which predict that turning on a switch will cause the lightbulb to emit light are more accurate that those who say that it's magic or mysterious. To quote Feynman: > We’ve learned from experience that the truth will out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature’s phenomena will agree or they’ll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven’t tried to be very careful in this kind of work. And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in Cargo Cult Science.
- maxxxxx 10y ago
  In physics or other hard sciences it's possible to validate a lot of models. I was more thinking about other areas like politics, management or software engineering issues. There it's often not possible to validate your model. It's even hard to tell if something goes well or not. And the feedback loop from decision to result can take years in many cases. So you pretty much are forced to do cargo culting.

kragen 10y ago

In CS (Systems, not Theory) research there's a big push now to move toward reproducibility by publishing code and dependencies; https://twitter.com/kragen/status/727220736633524228 https://twitter.com/kragen/status/727220736633524228 talks a bit about some of the historical context and details. Hackademia has a couple of synergistic efforts going on, directed mostly at securing the free-software distribution infrastructure and somewhat at simplifying system administration: on one hand you have reproducible builds being done by Tor, Mozilla, and Debian, and on the other hand you have Nix, Guix, and the like pushing purely-functional content-addressable package management. The upshot of all of this is that, in theory, in the near future, it will be practical to publish machine-executable instructions for reproducing the bit-identical executables and other output products you used in your research, at a cost of only a few gigabytes, so they can be exactly reproduced at any time in the future, and then modified to do new experiments; and it will be straightforward to make archival copies of this entire set of dependencies so that it won't be lost because some random guy died or decided to pull his modules from npm. Mike Hoye famously complained about how far we are from this in a Tweetstorm the other day: https://twitter.com/mhoye/status/725010388547330048 https://twitter.com/mhoye/status/725010388547330048 The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem. Nobody really knows what went wrong, but our best guess at this point is that the Sprite operating system had unknown bugs in it that gave a performance advantage to log-structured filesystems. (ChuckMcM may want to weigh in, since his company at the time (NetApp) was getting record-setting performance with WAFL, which is very similar to a LFS in some ways, but not others.) Jan Vitek, Jean Yang, Sam Tobin-Hochstadt, and Shriram Krishnamurthi are four prominent researchers who are advocating this change; they and others have created a new institution called "Artifact Evaluation Committees", which review the artifacts submitted by authors to see if they are able to reproduce the results the authors report. They've written an overview of it here, and they seem to already be getting significant results: http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-dont-publish-code.html http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-d... Hopefully over time this will create a bubble of real, reproducible science inside of the systems part of Computer "Science" research. [Earlier versions of this comment did not mention Krishnamurthi or Vitek and the synergistic work in hackademia, which was a combination of cluelessness and carelessness on my part. I regret the omissions.]

kirrent 10y ago

The parts that always strike me the most about this is that in 1974 Feynman lamented a lack of scientific rigour in deciding how we teach and how we manage crime and the criminal justice system. In both cases, despite 40 years having passed, it's hard to say there's been any large improvement in the application of rigour.

abecedarius 10y ago
There's some progress: https://en.wikipedia.org/wiki/Sequential_lineups https://en.wikipedia.org/wiki/Sequential_lineups The part that boggled me was how recent (and incomplete) was the introduction of blinding to police lineups. Apparently it really was (and is?) common for the witness to pick out a suspect, and for the cop to know the 'real' suspect and say things like "Are you sure? Take your time." From another Feynman talk: "This is not yet a scientific age."
naivepiano 10y ago
Do you have any link to that speech/article?

dschiptsov 10y ago

He probably could have been delighted to see all these "pendulums going in both directions at the same time" and other marvels of modern science we so often see on this site.

m1n1 10y ago

Ancient typos She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to __nut__ try to repeat psychological experiments, but only to change the conditions and see __hat__ happens.

Pica_soO 10y ago

The problem can not be solved by education alone. Education alone does not prevent the minds from making mistakes. Its about building and maintaining social structures that destroy cargo science and at the same time give freedom to groups/individuals who abstain from it. On envisions a credit system, in which you earn credibility by proofing something you did works- or by disproofing something that was wrong (obviously you get all credits that proof held). New credits enters the system with every entering scholar.