3 ms·

> ... doing the same thing twice is pointless. That's what psychologists say (according to Richard Feynman in his now-famous criticism of the practice in "Carg

by lutusp 14y ago

> ... doing the same thing twice is pointless. That's what psychologists say (according to Richard Feynman in his now-famous criticism of the practice in "Cargo Cult Science"), but it's a fundamentally flawed view of science. Replication -- an exact duplication of a study and hopefully its results -- is a cornerstone of science. And without full disclosure of the original study's method and results, replication isn't possible. > This is why the Hubble Constant started off being 150, then was 50, and now hovers around 69. Because everyone did the problem in their own unique way and didn't borrow each other's code and procedures. That isn't an argument for the practice of withholding methods, it's an argument against it. Arriving at a realistic value for the Hubble constant wasn't accelerated by withholding methods, it was slowed. Imagine Einstein claiming that mass and energy are bound together in a clearly defined relationship, but not publishing the relationship or its derivation. Arthur C. Clarke said, "Any sufficiently advanced technology is indistinguishable from magic." Apropos the present topic, a result unaccompanied by the methods used to produce it is also indistinguishable from magic, and certainly isn't science. > If I wrote it, I want first dibs on analysing it, I don't want someone else taking the results I created and publishing them before me. We're not taking about someone beating the originator to publication, we're talking about the originator publishing his methods along with his results and getting credit for both. > Politics is surprisingly omnipresent here. Perhaps, but always at the expense of science. Science relies on full disclosure -- both methods and results. How can a theoretical consensus be arrived at if the participants don't know what they're agreeing about?

J_Thomas 14y ago
"Replication -- an exact duplication of a study and hopefully its results -- is a cornerstone of science. And without full disclosure of the original study's method and results, replication isn't possible." My first quarter in grad school in statistics, a girl who was a couple of years ahead of me quit. She was doing an internship with an MD, and every time he got a new diagnosed kid in the project he pondered which group to put the kid in. "This one's going to die, where do I put him to make the results come out right?" The institution was considered second-class in that field, so he got the money to replicate a Harvard result. If he got the right results it would make him look more competent and he might get better grants. So he was doing his best to fudge the results so they would come out right. She tried to argue that he should do his statistics correctly and he disagreed. She was so upset that her work was pointless, and that her career would be pointless, that she quit completely. Ideally scientists would be rewarded for doing open science correctly. In some ways that is not the case now, and we should look at ways to fix that. How can average mediocre scientists get job security without keeping secrets? Perhaps we are giving too many people a chance to be professional scientists, so that too many of them must lose out?
- lutusp 14y ago
  What a sad account. What a negative outcome for science. And I'm sure it's common in many fields. > How can average mediocre scientists get job security without keeping secrets? By way of systematic changes, not by anything they necessarily can do themselves at the level you're describing. By way of people near the situation willing to speak out, like those who brought Jan Hendrik Schön down: http://en.wikipedia.org/wiki/Sch%C3%B6n_scandal http://en.wikipedia.org/wiki/Sch%C3%B6n_scandal Or in a similar way, by the actions of those who brought several highly regarded psychologists down: http://science.nbcnews.com/_news/2013/02/20/17032396-scandals-force-psychologists-to-do-some-soul-searching http://science.nbcnews.com/_news/2013/02/20/17032396-scandal... Title: "Scandals force psychologists to do some soul-searching" Quote relevant to the present topic: "For instance, right now there's no incentive for researchers to share their data, and a 2006 study found that of 141 researchers who had previously agreed to share their data, only 38 did so when asked." But the existence of the linked article and others like it means something is being done -- many psychologists have been outed as frauds, or fired, or forced into retirement. So there's reason to hope that, if enough daylight shines on these practices and laboratories, things will change.

by lutusp 14y ago

J_Thomas 14y ago

"Replication -- an exact duplication of a study and hopefully its results -- is a cornerstone of science. And without full disclosure of the original study's method and results, replication isn't possible." My first quarter in grad school in statistics, a girl who was a couple of years ahead of me quit. She was doing an internship with an MD, and every time he got a new diagnosed kid in the project he pondered which group to put the kid in. "This one's going to die, where do I put him to make the results come out right?" The institution was considered second-class in that field, so he got the money to replicate a Harvard result. If he got the right results it would make him look more competent and he might get better grants. So he was doing his best to fudge the results so they would come out right. She tried to argue that he should do his statistics correctly and he disagreed. She was so upset that her work was pointless, and that her career would be pointless, that she quit completely. Ideally scientists would be rewarded for doing open science correctly. In some ways that is not the case now, and we should look at ways to fix that. How can average mediocre scientists get job security without keeping secrets? Perhaps we are giving too many people a chance to be professional scientists, so that too many of them must lose out?

lutusp 14y ago
What a sad account. What a negative outcome for science. And I'm sure it's common in many fields. > How can average mediocre scientists get job security without keeping secrets? By way of systematic changes, not by anything they necessarily can do themselves at the level you're describing. By way of people near the situation willing to speak out, like those who brought Jan Hendrik Schön down: http://en.wikipedia.org/wiki/Sch%C3%B6n_scandal http://en.wikipedia.org/wiki/Sch%C3%B6n_scandal Or in a similar way, by the actions of those who brought several highly regarded psychologists down: http://science.nbcnews.com/_news/2013/02/20/17032396-scandals-force-psychologists-to-do-some-soul-searching http://science.nbcnews.com/_news/2013/02/20/17032396-scandal... Title: "Scandals force psychologists to do some soul-searching" Quote relevant to the present topic: "For instance, right now there's no incentive for researchers to share their data, and a 2006 study found that of 141 researchers who had previously agreed to share their data, only 38 did so when asked." But the existence of the linked article and others like it means something is being done -- many psychologists have been outed as frauds, or fired, or forced into retirement. So there's reason to hope that, if enough daylight shines on these practices and laboratories, things will change.