6 ms·

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous re

by s_q_b 10y ago

Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field. If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blame the QA engineer. It's my mistake, and so I must own it. These researchers need to do the same. I believe full publication of code and data should be required for peer-reviewed research. If I can't look at your raw data or your algorithm, it is simply not possible to determine whether the study is correct. The social sciences must adapt to more statistically rigorous methods. After all, the only two choices are adaptation or extinction.

jgalt212 10y ago
> sunlight all of their data and code This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.
- deleted 10y ago
  [deleted]
- WalterSear 10y ago
  Not directly relevant to the more widespread issue of how people should provide and react to criticism of scienctific work.
- lstamour 10y ago
  Presumably however other researchers could subscribe to the same source... third-party proprietary datasets, if non-partisan wrt research results, are still better than first-party proprietary datasets, right?
- Cerium 10y ago
  If the community demands the ability to review code and data then the usage of closed data sets will simply decrease until they comply. We should not base our standards of research on the ease of complying. We should do what we can to produce the most accurate results.
  - nickff 10y ago
    The problem is not one of convenience, the problem is that you are depriving yourself of huge amounts of valuable (closed) data, and this may cause large gaps in what is researchable.
    - eveningcoffee 10y ago
      This was the point. You have to pay the price for the principles and hope that the tide turns one day. If you take your suggested easy route then this day may never come.
      - nickff 10y ago
        My point is that there are large fields where we may not get access to open source data in the foreseeable future; in the meantime, we are depriving ourselves of potentially useful or valuable knowledge. Imagine if you could reduce the number of traffic deaths by analyzing a closed data set from Lyft or Uber; is this worth doing, or should we just tell the PhD candidate not to bother?
      - zmb_ 10y ago
        Are you a researcher and following those principles? Because it is very easy to demand other people pay the price for what you believe are the right principles. The reality of the way research funding and academia is set up (at least in Europe) is that the price you pay is likely your career. You can go through all the work and effort to follow those principles, and all that will happen is that the politicians will give the research funding and tenure positions to competitors that published more and told more exciting stories. Academia and research will necessarily reflect the incentives put in place by the people with the money (i.e., politicians). And right now they are giving money for exciting stories and publication counts. That is where the change must start.
        eveningcoffee 10y ago
        I did not demand anything. I just explained that when everybody complies, nothing will probably change.
- rwallace 10y ago
  Then it should be required whenever and to the extent it is possible. And where it is not possible, we should be considerably more skeptical of claims made, because trusting a claim based on a proprietary dataset requires a hefty dollop of faith.
  - Mendenhall 10y ago
    I agree, its like saying "I have proof I just cant show you".
- mcguire 10y ago
  And such data sets should be trusted...why?
- unimpressive 10y ago
  The problem is that a lot of studies don't suffer from this problem, and their data/code isn't open either. What we're looking at is a default-closed mindset, which needs to be inverted ASAP if these people want us to trust them in any significant capacity.
- ArkyBeagle 10y ago
  Then the provider of the subscription must be able to vouch for it or that data should not be used. The provider should also be .... sensitive to the need for that data to be used well, for any conclusion drawn from it to be reproducible.
- krapp 10y ago
  It's possible, it's just not legal. Someone could set up a Wikileaks type site for scientific and medical datasets and then what the law says becomes irrelevant.
hackuser 10y ago
> Good scientists should welcome this challenge with open arms They should welcome online trolling, abuse, and personal attacks? That is what Fiske criticizes in her article, not serious criticism and debate, which she explicitly welcomes and encourages.
- unimpressive 10y ago
  Fiske is pulling a classic Motte and Bailey strategy. http://slatestarcodex.com/2014/07/07/social-justice-and-words-words-words/ http://slatestarcodex.com/2014/07/07/social-justice-and-word... The Motte is that we should condemn personal attacks, trolling, and abuse. This is completely uncontroversial and nobody sane would disagree. The Bailey is that criticism made outside of the peer review process is somehow terrorism, which is so ludicrous that ones reaction should involve spirited laughter.
  - nickpsecurity 10y ago
    That was a great read. Illustrated the problems really well. I thought the antibiotic thing was hilarious. Still need more condensed posts to drop on anyone skeptical about the SJW effect happening where lines are drawn in prejudicial ways with words that are used as weapons. Many people might not go through such a long piece.
- wyager 10y ago
  Have you seen what passes for "online abuse" right now? The fact that Fisk earnestly uses the phrase "methodological terrorism" suggests that she is not to be trusted with negative connotations. Everyone agrees "abuse" (connotation: wife beating) is bad, so people like Fisk just redefine "abuse" to mean "being blunt online", and people who aren't inoculated against semantic overloading techniques fall for it.
- mafribe 10y ago
  They should welcome online trolling, abuse, and personal attacks? Nobody should welcome it, but -- as it comes with the territory -- they need to ignore it. She works on what is historically and cross-culturally arguably the most controversial subject, namely human sexuality. Attacks are to be expected, and the mature reaction to this is to lead by example: ignore the attacks and be as meticulously scientific as possible in the hope of inspiring others to be similarly mature. All the more so if you work in a field (psychology) that has a long history of unsupported claims and lack of methodological rigour. The reason science works, the reason we can fly to Mars, have a supercomputer in everybody's pocket today, can treat many diseases, is because sufficiently many adhere to a rigorous scientific methodology and every scientist is extremely critical of colleagues who don't. It is incumbent upon all scientists to hold themselves and others up to the highest standards. Computer science grew out of worries about the lack of rigor in the foundations of mathematics,triggered by the (re-)discovery of non-Euclidian geometry. Mathematics, the most rigorous of all scientific fields, is currently beginning to move towards formally verified proofs. To be even more reliable.
Someone 10y ago
"sunlight all of their data and code in a repository" Good idea where possible, but "all their data" will be problematic in many social science studies, and also in medicine, because of privacy laws.
- finid 10y ago
  Keep in mind that data can always be anonymized. So even most medical or social science research data can be put in a repo.
  - rando832 10y ago
    > Keep in mind that data can always be anonymized. No it can't. Some data can be anonymized, some can't.
    - RyanZAG 10y ago
      There's always a clever solution to anonymize any data. Post a description of the data and the problems with anonymizing it, and someone out there will always find a clever way to handle it. You don't have to solve everything yourself. That's the whole point of a peer driven scientific community.
  - dahart 10y ago
    Inferences in large data sets are more powerful than you think. Individual people have been identified from multiple "anonymized" scrubbed datasets, and its only getting easier as the datasets get bigger and the analytics tech gets better.
- wyager 10y ago
  Anonymize the data then, if plausible. We ought not to trust studies that are inherently irreproducible.
  - Someone 10y ago
    It's practically impossible to anonymize data without throwing away the information contained in it, doubly so if you don't know at the time what questions critics will want to see answered on that data.
merraksh 10y ago
As said toward the end of the article: We learn from our mistakes, but only if we recognize that they are mistakes. Debugging is a collaborative process. If you approve some code and I find a bug in it, I’m not an adversary, I’m a collaborator. If you try to paint me as an “adversary” in order to avoid having to correct the bug, that’s your problem.

by s_q_b 10y ago

jgalt212 10y ago

> sunlight all of their data and code This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.

deleted 10y ago
[deleted]
WalterSear 10y ago
Not directly relevant to the more widespread issue of how people should provide and react to criticism of scienctific work.
lstamour 10y ago
Presumably however other researchers could subscribe to the same source... third-party proprietary datasets, if non-partisan wrt research results, are still better than first-party proprietary datasets, right?
Cerium 10y ago
If the community demands the ability to review code and data then the usage of closed data sets will simply decrease until they comply. We should not base our standards of research on the ease of complying. We should do what we can to produce the most accurate results.
- nickff 10y ago
  The problem is not one of convenience, the problem is that you are depriving yourself of huge amounts of valuable (closed) data, and this may cause large gaps in what is researchable.
  - eveningcoffee 10y ago
    This was the point. You have to pay the price for the principles and hope that the tide turns one day. If you take your suggested easy route then this day may never come.
    - nickff 10y ago
      My point is that there are large fields where we may not get access to open source data in the foreseeable future; in the meantime, we are depriving ourselves of potentially useful or valuable knowledge. Imagine if you could reduce the number of traffic deaths by analyzing a closed data set from Lyft or Uber; is this worth doing, or should we just tell the PhD candidate not to bother?
    - zmb_ 10y ago
      Are you a researcher and following those principles? Because it is very easy to demand other people pay the price for what you believe are the right principles. The reality of the way research funding and academia is set up (at least in Europe) is that the price you pay is likely your career. You can go through all the work and effort to follow those principles, and all that will happen is that the politicians will give the research funding and tenure positions to competitors that published more and told more exciting stories. Academia and research will necessarily reflect the incentives put in place by the people with the money (i.e., politicians). And right now they are giving money for exciting stories and publication counts. That is where the change must start.
      - eveningcoffee 10y ago
        I did not demand anything. I just explained that when everybody complies, nothing will probably change.
rwallace 10y ago
Then it should be required whenever and to the extent it is possible. And where it is not possible, we should be considerably more skeptical of claims made, because trusting a claim based on a proprietary dataset requires a hefty dollop of faith.
- Mendenhall 10y ago
  I agree, its like saying "I have proof I just cant show you".
mcguire 10y ago
And such data sets should be trusted...why?
unimpressive 10y ago
The problem is that a lot of studies don't suffer from this problem, and their data/code isn't open either. What we're looking at is a default-closed mindset, which needs to be inverted ASAP if these people want us to trust them in any significant capacity.
ArkyBeagle 10y ago
Then the provider of the subscription must be able to vouch for it or that data should not be used. The provider should also be .... sensitive to the need for that data to be used well, for any conclusion drawn from it to be reproducible.
krapp 10y ago
It's possible, it's just not legal. Someone could set up a Wikileaks type site for scientific and medical datasets and then what the law says becomes irrelevant.

hackuser 10y ago

> Good scientists should welcome this challenge with open arms They should welcome online trolling, abuse, and personal attacks? That is what Fiske criticizes in her article, not serious criticism and debate, which she explicitly welcomes and encourages.

unimpressive 10y ago
Fiske is pulling a classic Motte and Bailey strategy. http://slatestarcodex.com/2014/07/07/social-justice-and-words-words-words/ http://slatestarcodex.com/2014/07/07/social-justice-and-word... The Motte is that we should condemn personal attacks, trolling, and abuse. This is completely uncontroversial and nobody sane would disagree. The Bailey is that criticism made outside of the peer review process is somehow terrorism, which is so ludicrous that ones reaction should involve spirited laughter.
- nickpsecurity 10y ago
  That was a great read. Illustrated the problems really well. I thought the antibiotic thing was hilarious. Still need more condensed posts to drop on anyone skeptical about the SJW effect happening where lines are drawn in prejudicial ways with words that are used as weapons. Many people might not go through such a long piece.
wyager 10y ago
Have you seen what passes for "online abuse" right now? The fact that Fisk earnestly uses the phrase "methodological terrorism" suggests that she is not to be trusted with negative connotations. Everyone agrees "abuse" (connotation: wife beating) is bad, so people like Fisk just redefine "abuse" to mean "being blunt online", and people who aren't inoculated against semantic overloading techniques fall for it.
mafribe 10y ago
They should welcome online trolling, abuse, and personal attacks? Nobody should welcome it, but -- as it comes with the territory -- they need to ignore it. She works on what is historically and cross-culturally arguably the most controversial subject, namely human sexuality. Attacks are to be expected, and the mature reaction to this is to lead by example: ignore the attacks and be as meticulously scientific as possible in the hope of inspiring others to be similarly mature. All the more so if you work in a field (psychology) that has a long history of unsupported claims and lack of methodological rigour. The reason science works, the reason we can fly to Mars, have a supercomputer in everybody's pocket today, can treat many diseases, is because sufficiently many adhere to a rigorous scientific methodology and every scientist is extremely critical of colleagues who don't. It is incumbent upon all scientists to hold themselves and others up to the highest standards. Computer science grew out of worries about the lack of rigor in the foundations of mathematics,triggered by the (re-)discovery of non-Euclidian geometry. Mathematics, the most rigorous of all scientific fields, is currently beginning to move towards formally verified proofs. To be even more reliable.

Someone 10y ago

"sunlight all of their data and code in a repository" Good idea where possible, but "all their data" will be problematic in many social science studies, and also in medicine, because of privacy laws.

finid 10y ago
Keep in mind that data can always be anonymized. So even most medical or social science research data can be put in a repo.
- rando832 10y ago
  > Keep in mind that data can always be anonymized. No it can't. Some data can be anonymized, some can't.
  - RyanZAG 10y ago
    There's always a clever solution to anonymize any data. Post a description of the data and the problems with anonymizing it, and someone out there will always find a clever way to handle it. You don't have to solve everything yourself. That's the whole point of a peer driven scientific community.
- dahart 10y ago
  Inferences in large data sets are more powerful than you think. Individual people have been identified from multiple "anonymized" scrubbed datasets, and its only getting easier as the datasets get bigger and the analytics tech gets better.
wyager 10y ago
Anonymize the data then, if plausible. We ought not to trust studies that are inherently irreproducible.
- Someone 10y ago
  It's practically impossible to anonymize data without throwing away the information contained in it, doubly so if you don't know at the time what questions critics will want to see answered on that data.

merraksh 10y ago

As said toward the end of the article: We learn from our mistakes, but only if we recognize that they are mistakes. Debugging is a collaborative process. If you approve some code and I find a bug in it, I’m not an adversary, I’m a collaborator. If you try to paint me as an “adversary” in order to avoid having to correct the bug, that’s your problem.