5 ms·

Replications often don't happen, because it's not considered "new," and who wants to pay for something that's not new? Who can get accolades and titles and jour

Replications often don't happen, because it's not considered "new," and who wants to pay for something that's not new? Who can get accolades and titles and journal articles and tenure and HIGHER IMPACT FACTORS out of the drudgery of replication? There are a lot of science popularizers out there who focus a lot on the power of science, and its ability to self-correct. And given their audience, it's probably a good message to get out there. It's a lot better than most other approaches we've had throughout history. But I think they can give the wrong impression of science (often spoken of as SCIENCE!!!! with exclamation marks) as this process delivered to humanity on stone tablets from on high, and with a little too much trust in the idea that "the arc of science is long but it bends toward correctness." Science is a frustratingly, painfully, agonizingly human process, made up of people in various organizations, with their own lives and goals and objectives. It's a process where there's been enough of a correlation between what its participants want, and what accuracy/truth/improvement demands, that it's worked pretty well so far, and it gets good results eventually. But there are also perverse incentives in organizations and personal failings in its participants. Things like funders' quest for short-term results, the desire to only back proven winners, the drive to publish or perish, etc etc. Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." I guess it's a little like the slow, slow process of natural selection. Given enough time, you can get some pretty impressive, complex stuff out of it, but you can also get stuck in a local maximum for millions of years. I don't know what the solution is. Perhaps it would be nice for graduation requirements for master's/PhD programs to require attempted replication of some existing works -- but then there's the risk of selecting things that are easy to replicate. Or maybe some mandatory percentage of research grants going toward a common replication pool. Setting up a good incentives structure is hard, because any institution that constrains itself by following new rules "for the good of science" will find itself falling behind in comparison with institutions who only work on the new exciting stuff and get more of the media attention. I feel fortunate that I'm in computer science, where a lot of our work is inherently easily replicable due to the nature of code (of course, once it gets into user studies there's so much wiggle room that the same problems arise), and I always appreciate when a research group releases testable, compilable, documented code alongside their papers. I'd like to see that sort of practice be more mandatory in CS, but when official prestige comes from journals only, the incentive structures don't reward CS researchers for making maintainable, understandable code -- just code that gets the results and pretty pictures. Academic code is infamous for being poorly-architected. ---- Along some of these lines, it's always a good read to go through Feynman's "Cargo Cult Science": http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html

astazangasta 11y ago
There is a profound problem with the scientific process that I have been worrying about that rarely gets mentioned: The appeal and power of science is that you propose a model and then test it with evidence. With strong evidence you can validate the predictive efficacy of your model. However, the model is not "true", it is only predictive. Furthermore, it's only predictive to the extent that the evidence backs it up. The problem with this process is that it hangs entirely on the evidence. The models themselves do NOT emerge from the scientific process; they emerge from whatever - prejudice, confabulation, logic, supposition, etc. In other words, the original conjecture of a scientific model is merely a product of the pre-existing beliefs of the investigator. Coupling poor evidence to this sort of conjecture produces "science" that is simply supposition or bias with a veneer of credibility. Given the widespread abuse of methodology (p-hacking, poor understanding of statistics, small or unrepresentative sample sizes, not reporting negative experiments, bad experimental design) this means a lot of "science" is merely varnished prejudice.
- dj-wonk 11y ago
  I don't see this as a problem. Perhaps I define some concepts differently. As I see it, the scientific method is a good way to test a model against reality. It is less suited to be a "generative" process for making models. (There are other ways to test models too: internal consistency, interpretability, and so on.) Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. I'm not saying one could not using more "technical" or "rigorous" ways to construct models -- there are, of course, such methods. I'm just saying that "science" (in the Chalmers sense) is about mostly about testability of a model against evidence.
  - astazangasta 11y ago
    >Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. These are all very positive. How about: racism, misogyny, bias, jealousy, and stubbornness. These, also, can feed into the creation of theories, and do. And, again, the point is that this process is only as good as the evidence; when the evidence is lacking, we're left with only the investigator's expertise to justify the model. That's not science.

astazangasta 11y ago

There is a profound problem with the scientific process that I have been worrying about that rarely gets mentioned: The appeal and power of science is that you propose a model and then test it with evidence. With strong evidence you can validate the predictive efficacy of your model. However, the model is not "true", it is only predictive. Furthermore, it's only predictive to the extent that the evidence backs it up. The problem with this process is that it hangs entirely on the evidence. The models themselves do NOT emerge from the scientific process; they emerge from whatever - prejudice, confabulation, logic, supposition, etc. In other words, the original conjecture of a scientific model is merely a product of the pre-existing beliefs of the investigator. Coupling poor evidence to this sort of conjecture produces "science" that is simply supposition or bias with a veneer of credibility. Given the widespread abuse of methodology (p-hacking, poor understanding of statistics, small or unrepresentative sample sizes, not reporting negative experiments, bad experimental design) this means a lot of "science" is merely varnished prejudice.

dj-wonk 11y ago
I don't see this as a problem. Perhaps I define some concepts differently. As I see it, the scientific method is a good way to test a model against reality. It is less suited to be a "generative" process for making models. (There are other ways to test models too: internal consistency, interpretability, and so on.) Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. I'm not saying one could not using more "technical" or "rigorous" ways to construct models -- there are, of course, such methods. I'm just saying that "science" (in the Chalmers sense) is about mostly about testability of a model against evidence.
- astazangasta 11y ago
  >Iteration, intuition, creativity, variation, intelligence, luck, guesswork, and more are what create theories. These are all very positive. How about: racism, misogyny, bias, jealousy, and stubbornness. These, also, can feed into the creation of theories, and do. And, again, the point is that this process is only as good as the evidence; when the evidence is lacking, we're left with only the investigator's expertise to justify the model. That's not science.