8 ms·

Science Isn’t Broken

74 points by ot 11y ago

dluan 11y ago
I think it'd be cool if there was a rogue batman scientist on the loose, self or publicly funded, who'd go around cleaning up the streets by replicating and forcing retractions. Why batman? Because he had to fight cops just as often as baddies. Not that the current system is necessarily corrupt, but that our current system is almost as blind when it comes to recognition of good science. Our current system is a bit, well, dumb.
- gd1 11y ago
  http://climateaudit.org/ http://climateaudit.org/
  - dluan 11y ago
    We should organize a rogue scientist justice league.
    - tdaltonc 11y ago
      I'm in. I can do cognitive-neuro, psych, and experimental-econ.
  - gjm11 11y ago
    Much narrower focus than dluan's proposal (climate science only); pretty much entirely one-sided (criticizes work supporting the consensus view only, kinda like if Batman only fought the cops). The guy who runs it isn't a scientist (though you could argue that he's a statistician, which might be about as good for dluan's purposes). So I don't think this really fits dluan's description very well.
- plonh 11y ago
  Retraction Watch is close.
- kansface 11y ago
  You'd need a rogue in each specialty - I'm also guessing that replicating studies doesn't lead to tenure or papers in Nature.
  - dluan 11y ago
    Batman doesn't care about tenure.
  - cossatot 11y ago
    Nature editors LOVE controversy though. In my experience they're not really interested in evaluating the long-term substance of a paper as much as if it causes (or pretends to end[1]) a fight right now. [1] It's really hard to end a fight in 4 pages.
WalterSear 11y ago
That made a pretty convincing argument that it's broken, followed by thoroughly unconvincing, hand-wavey, claim to victory because "we eventually muddle through."
thinkmoore 11y ago
I feel like every description of the problem of p-values I read is missing the real issue. And that's that people are using the p-value incorrectly. A fairly standard definition of the p-value (from wikipedia) says: "In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as α." What this description is missing though is the crucial importance of the fact that the threshold value is chosen before doing the analysis. And moreover, that the entire analysis plan has been chosen before doing the analysis. Because what the p-value is really telling you is the probability that on repeating the experiment (and its accompanying analysis!) you would see a result as or more extreme than what you observed. If your experiment comprises "try all of the combinations of variables to see what gives me the best answer", the p-value you compute would need to be some very fancy test that took that into account... and you would see your analysis as having much less statistical power. For a simple example, look at a statistically rigorous method for dealing with multiple hypothesis testing when you plan it in advance: https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction. Of course p-hacking is bad. The problem isn't frequentist statistics or p-values though, its scientists not understanding the statistics that they use. If you want to use a p-value to help make a decision about a hypothesis, you have to commit to your analysis plan in advance. edit: Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose. To sum up: when you have people who have no idea what they're doing do statistics, they will do it badly.
- Hermel 11y ago
  > Because what the p-value is really telling you is the probability that on repeating the experiment you would see a result as or more extreme than what you observed. That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes). This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".
Xcelerate 11y ago
I think there needs to be a classification system for the "rigor" of studies. A low rigor rating doesn't mean bad science; it just means that the topic is very complex and difficult to study. For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor". Studies that assess whether watermelon is linked to heart disease would be "low rigor". The question now is how to come up with a rigorous classification system...
- Osmium 11y ago
  > For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor". You mean like, say, measuring the elementary charge of an electron to high precision? ;) Richard Feynman has something to say about the rigor of those experiments... > We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. > Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that... [from "Surely You're Joking, Mr. Feynman!" via Wikipedia–I know this isn't exactly what you were talking about in your comment, but I thought it makes for an interesting discussion regardless] Interesting paper on this here: http://arxiv.org/pdf/physics/0508199v1.pdf http://arxiv.org/pdf/physics/0508199v1.pdf and also another example here with the Hubble constant here: http://www.pnas.org/content/101/1/8/F2.expansion.html http://www.pnas.org/content/101/1/8/F2.expansion.html Even in Mullikan's case, the paper I linked seems to suggest he might have intentionally omitted data that didn't fit his narrative (I only know what I've read in that paper, so I wouldn't like to make any stronger claims than that). Whether he was 'right' to do that is an interesting question. Clearly the oil drop experiment was ground-breaking, and the results hugely important, but omitting inconvenient results is clearly a huge ethical violation. However, his intuition was correct, and he was observing a real effect, and the measurements he kept were good. So who can say? Science seems to be about asking the right questions as much as finding the right answers. Sometimes good science is just to get smart people talking about the "right" things, and the correct answers come later. Perhaps peripheral, but another favourite example of mine is the Lennard-Jones potential for modelling e.g. noble gases. It has an attractive term and a repulsive term. The attractive term is distance to the power 6 and based on good physical reasoning; the repulsive term is distance to the power 12 and entirely arbitrary, just because it was easier to calculate since you'd already calculated the power 6 term. So, in a sense, it was completely made up. But it was still a useful tool because it helped people start asking the right questions, and exploring these systems, even if it was, in a sense, wrong.
1971genocide 11y ago
Science and modern capitalistic thinking do not work well together. Most people in business view any sort of rigorous research as a "cost" center. And this thinking has diffused into government too. On the other hand marketing and advertising produces the most value for dollar. Why ? In a market those activities generate new demand - something that a capitastic system needs in order to survive. Why spend 20 years doing research while we could spend 1 year aggressively expanding our market. Even a 1% growth would he worth it given a 0% growth if the money was spent doing research. I like that rather than villify a single person Nate silver points the finger at the system. This type of thinking is where we should all be heading towards.
- theophrastus 11y ago
  It is worrisome that you were here downvoted for this view; as it is one that I believe (as a working scientist) is one of the larger issues that has been left out of this discussion. For example, public universities used to be fairly unconstrained in their support of grant research. Now most, if not all, have an office devoted to the commercialization (patent protection) of their own publicly supported research; quite often subsidized by private interests. So if a researcher wanted to study the effects of cell-phone usage on the developing brain, (which didn't involve me, but rather a colleague), then suddenly several large companies were going to "lamentably" pull out of supporting the athletic department with its "amateur sports" endeavors. And the list of subtle control of "public" research continues. Science and commercialism can mix productively, but it should never be done covertly. Unfortunately corporate thinking is nearly always covert.
- brc 11y ago
  On the contrary, only free markets generate enough surplus capital for it to be invested in long term research. Planned and managed economies don't generate enough production to even feed and supply their existing populations well. It's a false dichotomy to say that you can only choose between 20 year research and 1 year marketing. Plenty of private money goes into research with long term payoff. There are private companies engaged in fusion research, because the payoff is huge if it is achieved. Even the public money that goes into publicly funded research comes from private earnings in the first place. So science and free markets are absolutely suited to each other. That's before we even start discussing corruption of objectives in totalitarian states, based on leaders whims (ie, Nazi science research, Lysenko etc)
joaorico 11y ago
I like this summary of the evidence that most published research is false[1]: -"One of the hottest topics in science has two main conclusions: Most published research is false There is a reproducibility crisis in science The first claim is often stated in a slightly different way: that most results of scientific experiments do not replicate." - [Ioannidis'] "Paper: Why most published research findings are false. Main idea: People use hypothesis testing to determine if specific scientific discoveries are significant. This significance calculation is used as a screening mechanism in the scientific literature. Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation." - then it summarises in the same way 7 other papers, including the Many Labs results. -"I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely We need more statistical literacy We need more computational literacy We need to require code be published We need mechanisms of peer review that deal with code We need a culture that doesn't use reproducibility as a weapon We need increased transparency in review and evaluation of papers" - the final paragraph: "The Many Labs results suggest that the hype about the failures of science are, at the very least, premature. I think an equally important idea is that science has pretty much always worked with some number of false positive and irreplicable studies. This was beautifully described by Jared Horvath in this blog post from the Economist. I think the take home message is that regardless of the rate of false discoveries, the scientific process has led to amazing and life-altering discoveries." [1] http://simplystatistics.org/2013/12/16/a-summary-of-the-evidence-that-most-published-research-is-false/ http://simplystatistics.org/2013/12/16/a-summary-of-the-evid...
speechduh 11y ago
I saw a great talk about this @ SIAM data mining by Dave Madigan, for http://omop.org/ http://omop.org/ Basically, they took a TON of demographic research in the health sciences, explored all possible hyperparameter tunings, and found that they could get p-values of 0.05 in either direction for most of the papers depending on choice of data sources and many other types of hyperparameters.
jonasvp 11y ago
From the article: > some people have begun to ask: “Is science broken?” > I’ve spent many months asking dozens of scientists this question, and the answer I’ve found is a resounding no. News at 11: cognitive dissonance is a thing.
- robotresearcher 11y ago
  Science's old and new fruits benefit the lives of almost everyone on the planet every day and revolutionary new technologies, therapies and knowledge are created faster than ever before. It's accelerating, not slowing down and certainly not stopping. So it's not very broken. Things can certainly improve, and they usually do, perhaps a bit gradually. Take a long view. Science is funded by the public and not the church these days for example. But 'broken'? Not really. Edit: and non-scientists are very quick to think they know the problems and offer solutions. Working scientists are actively discussing the problems and trying things out all the time. These are not the dumbest people you'll meet. The current model is a bit like democracy - it's the worst system apart from all the other things we've ever tried. To be clear: I'm not an apologist and I'm not sweeping troubles under the rug. It's just that it is complicated, we're working on it, and it is currently much too productive to be called broken. We have to be careful not to break it.
  - tdaltonc 11y ago
    We don't expect insiders to objectively improve most social systems (politics, business, education, defense, etc). Why should we expect scientists to fix science (as opposed to just pushing the system to serve the insiders)?
    - robotresearcher 11y ago
      We do expect insiders to improve those systems. Why would outsiders know what needs done?
      - WalterSear 11y ago
        We would like them to, but it's reasonable not to expect them to.
yk 11y ago
Science is not broken, but the scientific process is severely misrepresented. I can only really speak about physics, but at least there and I believe also in other fields, people do experiments, observations etc. and then write papers on whatever they did. The papers are intended for a highly specialized audience and contain a lot of detail besides some claim and a p value and there are usually quite a few closely related papers. Actually I sat in more than one conference session where every single talk had the title "The X-ray spectra of NGC XXXX." Then someone will write a review article on the field. And that person is highly specialized, has probably read hundreds of papers in the field over the last decade(s) and has hands on experience. That guy knows that if there is no mention of X in the methodology section, then the authors did not deal with X, which may or may not be justifiable. After that someone writes a textbook, that guy will read several review papers, and he has probably enough experience to know that a sentence like "compare the contradictory results of Y" is a indication that he should look into the actual papers and perhaps use very cautionary language or omit the topic entirely. At that point, when it appears in textbooks, a result is settled science, not when the first indications appear in a paper. On the other hand, science writes want a headline "Something baffles scientists." Well, the scientists are probably baffled because it contradicts their intuition, which is a strong indicator that something is not really working. This something may be a honest mistake by the authors, a poorly understood experimental effect, outright fraud or it may be something worth reporting. The entire process of science writing is geared towards the papers which are most likely to be wrong.
- WalterSear 11y ago
  IMHO, the process is misrepresented >and< the process is broken. I think the article does a good job of explaining many (but not all) of the reasons I think it is broken.
- Retric 11y ago
  I agree with what your saying. However, I feel publish or perish has infected many fields resulting in a lot of crap instead of more focused high quality work. Few people can risk spending 10 years on something and have nothing to show for it. It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.
unabst 11y ago
Science might not be broken for the reasons mentioned, but I find the article overlooks the greatest evidence of all which is the article itself. Objectively deconstructing the practice of p-hacking is science at work and a scientific work. So say, if we devised a way to measure p-value based research for it's p-hack-ability, we would have a way to validate what that research is saying (or not saying), as well as test the integrity of the research and its researchers.
animefan 11y ago
This article is a good summary of the issues of the (mis)use of statistics in science. Having done some research myself, in most cases the statistics don't scream a particular message to you, and it's really hard to understand the data. Even without any external pressures, statistical tools like p-values have limits. If you run 10 different models (all a priori reasonable) and 8 of them seem to say roughly the same thing, does that mean your result is correct?

74 points by ot 11y ago

dluan 11y ago

I think it'd be cool if there was a rogue batman scientist on the loose, self or publicly funded, who'd go around cleaning up the streets by replicating and forcing retractions. Why batman? Because he had to fight cops just as often as baddies. Not that the current system is necessarily corrupt, but that our current system is almost as blind when it comes to recognition of good science. Our current system is a bit, well, dumb.

gd1 11y ago
http://climateaudit.org/ http://climateaudit.org/
- dluan 11y ago
  We should organize a rogue scientist justice league.
  - tdaltonc 11y ago
    I'm in. I can do cognitive-neuro, psych, and experimental-econ.
- gjm11 11y ago
  Much narrower focus than dluan's proposal (climate science only); pretty much entirely one-sided (criticizes work supporting the consensus view only, kinda like if Batman only fought the cops). The guy who runs it isn't a scientist (though you could argue that he's a statistician, which might be about as good for dluan's purposes). So I don't think this really fits dluan's description very well.
plonh 11y ago
Retraction Watch is close.
kansface 11y ago
You'd need a rogue in each specialty - I'm also guessing that replicating studies doesn't lead to tenure or papers in Nature.
- dluan 11y ago
  Batman doesn't care about tenure.
- cossatot 11y ago
  Nature editors LOVE controversy though. In my experience they're not really interested in evaluating the long-term substance of a paper as much as if it causes (or pretends to end[1]) a fight right now. [1] It's really hard to end a fight in 4 pages.

WalterSear 11y ago

That made a pretty convincing argument that it's broken, followed by thoroughly unconvincing, hand-wavey, claim to victory because "we eventually muddle through."

thinkmoore 11y ago

I feel like every description of the problem of p-values I read is missing the real issue. And that's that people are using the p-value incorrectly. A fairly standard definition of the p-value (from wikipedia) says: "In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as α." What this description is missing though is the crucial importance of the fact that the threshold value is chosen before doing the analysis. And moreover, that the entire analysis plan has been chosen before doing the analysis. Because what the p-value is really telling you is the probability that on repeating the experiment (and its accompanying analysis!) you would see a result as or more extreme than what you observed. If your experiment comprises "try all of the combinations of variables to see what gives me the best answer", the p-value you compute would need to be some very fancy test that took that into account... and you would see your analysis as having much less statistical power. For a simple example, look at a statistically rigorous method for dealing with multiple hypothesis testing when you plan it in advance: https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction. Of course p-hacking is bad. The problem isn't frequentist statistics or p-values though, its scientists not understanding the statistics that they use. If you want to use a p-value to help make a decision about a hypothesis, you have to commit to your analysis plan in advance. edit: Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose. To sum up: when you have people who have no idea what they're doing do statistics, they will do it badly.

Hermel 11y ago
> Because what the p-value is really telling you is the probability that on repeating the experiment you would see a result as or more extreme than what you observed. That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes). This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".

Xcelerate 11y ago

I think there needs to be a classification system for the "rigor" of studies. A low rigor rating doesn't mean bad science; it just means that the topic is very complex and difficult to study. For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor". Studies that assess whether watermelon is linked to heart disease would be "low rigor". The question now is how to come up with a rigorous classification system...

Osmium 11y ago
> For instance, studies that compute the gyromagnetic ratio of the electron (to 10 decimal places no less) and then compare that against the experimentally obtained value would be classified as "high rigor". You mean like, say, measuring the elementary charge of an electron to high precision? ;) Richard Feynman has something to say about the rigor of those experiments... > We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. > Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that... [from "Surely You're Joking, Mr. Feynman!" via Wikipedia–I know this isn't exactly what you were talking about in your comment, but I thought it makes for an interesting discussion regardless] Interesting paper on this here: http://arxiv.org/pdf/physics/0508199v1.pdf http://arxiv.org/pdf/physics/0508199v1.pdf and also another example here with the Hubble constant here: http://www.pnas.org/content/101/1/8/F2.expansion.html http://www.pnas.org/content/101/1/8/F2.expansion.html Even in Mullikan's case, the paper I linked seems to suggest he might have intentionally omitted data that didn't fit his narrative (I only know what I've read in that paper, so I wouldn't like to make any stronger claims than that). Whether he was 'right' to do that is an interesting question. Clearly the oil drop experiment was ground-breaking, and the results hugely important, but omitting inconvenient results is clearly a huge ethical violation. However, his intuition was correct, and he was observing a real effect, and the measurements he kept were good. So who can say? Science seems to be about asking the right questions as much as finding the right answers. Sometimes good science is just to get smart people talking about the "right" things, and the correct answers come later. Perhaps peripheral, but another favourite example of mine is the Lennard-Jones potential for modelling e.g. noble gases. It has an attractive term and a repulsive term. The attractive term is distance to the power 6 and based on good physical reasoning; the repulsive term is distance to the power 12 and entirely arbitrary, just because it was easier to calculate since you'd already calculated the power 6 term. So, in a sense, it was completely made up. But it was still a useful tool because it helped people start asking the right questions, and exploring these systems, even if it was, in a sense, wrong.

1971genocide 11y ago

Science and modern capitalistic thinking do not work well together. Most people in business view any sort of rigorous research as a "cost" center. And this thinking has diffused into government too. On the other hand marketing and advertising produces the most value for dollar. Why ? In a market those activities generate new demand - something that a capitastic system needs in order to survive. Why spend 20 years doing research while we could spend 1 year aggressively expanding our market. Even a 1% growth would he worth it given a 0% growth if the money was spent doing research. I like that rather than villify a single person Nate silver points the finger at the system. This type of thinking is where we should all be heading towards.

theophrastus 11y ago
It is worrisome that you were here downvoted for this view; as it is one that I believe (as a working scientist) is one of the larger issues that has been left out of this discussion. For example, public universities used to be fairly unconstrained in their support of grant research. Now most, if not all, have an office devoted to the commercialization (patent protection) of their own publicly supported research; quite often subsidized by private interests. So if a researcher wanted to study the effects of cell-phone usage on the developing brain, (which didn't involve me, but rather a colleague), then suddenly several large companies were going to "lamentably" pull out of supporting the athletic department with its "amateur sports" endeavors. And the list of subtle control of "public" research continues. Science and commercialism can mix productively, but it should never be done covertly. Unfortunately corporate thinking is nearly always covert.
brc 11y ago
On the contrary, only free markets generate enough surplus capital for it to be invested in long term research. Planned and managed economies don't generate enough production to even feed and supply their existing populations well. It's a false dichotomy to say that you can only choose between 20 year research and 1 year marketing. Plenty of private money goes into research with long term payoff. There are private companies engaged in fusion research, because the payoff is huge if it is achieved. Even the public money that goes into publicly funded research comes from private earnings in the first place. So science and free markets are absolutely suited to each other. That's before we even start discussing corruption of objectives in totalitarian states, based on leaders whims (ie, Nazi science research, Lysenko etc)

joaorico 11y ago

I like this summary of the evidence that most published research is false[1]: -"One of the hottest topics in science has two main conclusions: Most published research is false There is a reproducibility crisis in science The first claim is often stated in a slightly different way: that most results of scientific experiments do not replicate." - [Ioannidis'] "Paper: Why most published research findings are false. Main idea: People use hypothesis testing to determine if specific scientific discoveries are significant. This significance calculation is used as a screening mechanism in the scientific literature. Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation." - then it summarises in the same way 7 other papers, including the Many Labs results. -"I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely We need more statistical literacy We need more computational literacy We need to require code be published We need mechanisms of peer review that deal with code We need a culture that doesn't use reproducibility as a weapon We need increased transparency in review and evaluation of papers" - the final paragraph: "The Many Labs results suggest that the hype about the failures of science are, at the very least, premature. I think an equally important idea is that science has pretty much always worked with some number of false positive and irreplicable studies. This was beautifully described by Jared Horvath in this blog post from the Economist. I think the take home message is that regardless of the rate of false discoveries, the scientific process has led to amazing and life-altering discoveries." [1] http://simplystatistics.org/2013/12/16/a-summary-of-the-evidence-that-most-published-research-is-false/ http://simplystatistics.org/2013/12/16/a-summary-of-the-evid...

speechduh 11y ago

I saw a great talk about this @ SIAM data mining by Dave Madigan, for http://omop.org/ http://omop.org/ Basically, they took a TON of demographic research in the health sciences, explored all possible hyperparameter tunings, and found that they could get p-values of 0.05 in either direction for most of the papers depending on choice of data sources and many other types of hyperparameters.

jonasvp 11y ago

From the article: > some people have begun to ask: “Is science broken?” > I’ve spent many months asking dozens of scientists this question, and the answer I’ve found is a resounding no. News at 11: cognitive dissonance is a thing.

robotresearcher 11y ago
Science's old and new fruits benefit the lives of almost everyone on the planet every day and revolutionary new technologies, therapies and knowledge are created faster than ever before. It's accelerating, not slowing down and certainly not stopping. So it's not very broken. Things can certainly improve, and they usually do, perhaps a bit gradually. Take a long view. Science is funded by the public and not the church these days for example. But 'broken'? Not really. Edit: and non-scientists are very quick to think they know the problems and offer solutions. Working scientists are actively discussing the problems and trying things out all the time. These are not the dumbest people you'll meet. The current model is a bit like democracy - it's the worst system apart from all the other things we've ever tried. To be clear: I'm not an apologist and I'm not sweeping troubles under the rug. It's just that it is complicated, we're working on it, and it is currently much too productive to be called broken. We have to be careful not to break it.
- tdaltonc 11y ago
  We don't expect insiders to objectively improve most social systems (politics, business, education, defense, etc). Why should we expect scientists to fix science (as opposed to just pushing the system to serve the insiders)?
  - robotresearcher 11y ago
    We do expect insiders to improve those systems. Why would outsiders know what needs done?
    - WalterSear 11y ago
      We would like them to, but it's reasonable not to expect them to.

yk 11y ago

Science is not broken, but the scientific process is severely misrepresented. I can only really speak about physics, but at least there and I believe also in other fields, people do experiments, observations etc. and then write papers on whatever they did. The papers are intended for a highly specialized audience and contain a lot of detail besides some claim and a p value and there are usually quite a few closely related papers. Actually I sat in more than one conference session where every single talk had the title "The X-ray spectra of NGC XXXX." Then someone will write a review article on the field. And that person is highly specialized, has probably read hundreds of papers in the field over the last decade(s) and has hands on experience. That guy knows that if there is no mention of X in the methodology section, then the authors did not deal with X, which may or may not be justifiable. After that someone writes a textbook, that guy will read several review papers, and he has probably enough experience to know that a sentence like "compare the contradictory results of Y" is a indication that he should look into the actual papers and perhaps use very cautionary language or omit the topic entirely. At that point, when it appears in textbooks, a result is settled science, not when the first indications appear in a paper. On the other hand, science writes want a headline "Something baffles scientists." Well, the scientists are probably baffled because it contradicts their intuition, which is a strong indicator that something is not really working. This something may be a honest mistake by the authors, a poorly understood experimental effect, outright fraud or it may be something worth reporting. The entire process of science writing is geared towards the papers which are most likely to be wrong.

WalterSear 11y ago
IMHO, the process is misrepresented >and< the process is broken. I think the article does a good job of explaining many (but not all) of the reasons I think it is broken.
Retric 11y ago
I agree with what your saying. However, I feel publish or perish has infected many fields resulting in a lot of crap instead of more focused high quality work. Few people can risk spending 10 years on something and have nothing to show for it. It's a negative feedback loop where quality is hard for people outside the field to judge. So, it's just race to see who can lower quality more while still being published.

unabst 11y ago

Science might not be broken for the reasons mentioned, but I find the article overlooks the greatest evidence of all which is the article itself. Objectively deconstructing the practice of p-hacking is science at work and a scientific work. So say, if we devised a way to measure p-value based research for it's p-hack-ability, we would have a way to validate what that research is saying (or not saying), as well as test the integrity of the research and its researchers.

animefan 11y ago

This article is a good summary of the issues of the (mis)use of statistics in science. Having done some research myself, in most cases the statistics don't scream a particular message to you, and it's really hard to understand the data. Even without any external pressures, statistical tools like p-values have limits. If you run 10 different models (all a priori reasonable) and 8 of them seem to say roughly the same thing, does that mean your result is correct?