5 ms·

CICLing : Computational Linguistics and Intelligent Text Processing (Natural Language Processing) conference [0] uses an open data/software method to eliminate

CICLing : Computational Linguistics and Intelligent Text Processing (Natural Language Processing) conference [0] uses an open data/software method to eliminate such frauds. They have a Verifiability, reproducibility, and working description policy which reads as follows Starting from 2011, CICLing implements a policy of giving preference to papers with verifiable and reproducible results: If the authors claim to have obtained some results, we encourage them to make all the input data necessary to verify and reproduce the results available to the community. If the authors claim to advance human knowledge by introducing an algorithm, we encourage them to make the algorithm itself -- and not only its (usually vague and incomplete) description -- available to the public. We do not require any demo or tool based on your paper, but instead a form of proof and working description of the algorithm in addition to the verbal description given in your paper. An approximation of the idea is the code submitted with Church & Umemura's paper to be permanently hosted at CICLing servers, and cited in the paper (see last line): you see, we don't mean anything complicated. Obviously, you are also encouraged to show demo programs or tools based on your method, either as part of your talk or (better) at the demo session, and we will also be happy to host on our servers such software that complements your paper. However, this is not required. In contrast, we do believe that a publicly published scientific paper must be accompanied by a minimal working description of the algorithm, open-source and available to the community. Also, we do not ask for impossible: if you present a large system, especially commercially distributed or a property of your company, then we do not expect you to provide the system. Our point is that when the software and data can be provided, it should be provided. We do not yet have specific rules: we hope to elaborate the rules basing on this year's experience, so please use common sense. See the problems this policy is to address, as well as the list of software reviewing committee and instructions for the reviewers. [0] http://www.cicling.org/2013/ http://www.cicling.org/2013/

roel_v 14y ago
Right, and which is perfectly useless to prevent what Stapel did. "We encourage them to make <etc>"? Instead of spending 5 days fabricating coherent data, you spend 6 - an additional day to come up with a plausible reason for not having to divulge raw data. I'm not a proponent of requiring everybody to submit their lab notes with every paper submission, in fact I have no credible solution to the whole problem at all - but feel-good policies like this are no more than something for editors of a journal to point at later when fraud is found, to be able to say "yeah but we did something" (i.e., the classical politician's fallacy).
- denzil_correa 14y ago
  > Right, and which is perfectly useless to prevent what Stapel did. Nothing of that sort is claimed; just because you do not have a fool proof method doesn't mean you take limited steps to stop it. The least you can do is put hygiene checks. Most reputed conferences wouldn't even do that. Also, how and why is this a feel-good policy? The program committee of the conference is reputed enough to check for fraud/discrepancies. After all, the Stapel fraud (like many others) was detected by his peers. FYI - this is not a journal either.
  - jhdevos 14y ago
    You used the phrase 'to eliminate such frauds.' The suggestion (to me) is that this is a fool proof method that eliminates such frauds. I would contend this is not true; fraud is still relatively easy. The question is how much this promotes a false sense of security. Stapel's fraud was only detected after many years, and after his results had been widely published. I still hear some of his conclusions quoted now and then as if it were established truth, by people who have not yet realised that it is one of the famous Stapel ones.
    - denzil_correa 14y ago
      Some clarifications - + No method/suggestion is fool proof in the world of publications; simply can't be + Stapel's fraud was "data fraud" which is eliminated if your work on publicly available datasets + NLP community encourages work on public data and you can't get Stapel level of recognition if you do not have your data/tools out in the open + Any suggestion/method if not fool proof can be said to promote a false sense of security; it is important to start somewhere rather than maintain status quo

roel_v 14y ago

Right, and which is perfectly useless to prevent what Stapel did. "We encourage them to make <etc>"? Instead of spending 5 days fabricating coherent data, you spend 6 - an additional day to come up with a plausible reason for not having to divulge raw data. I'm not a proponent of requiring everybody to submit their lab notes with every paper submission, in fact I have no credible solution to the whole problem at all - but feel-good policies like this are no more than something for editors of a journal to point at later when fraud is found, to be able to say "yeah but we did something" (i.e., the classical politician's fallacy).

denzil_correa 14y ago
> Right, and which is perfectly useless to prevent what Stapel did. Nothing of that sort is claimed; just because you do not have a fool proof method doesn't mean you take limited steps to stop it. The least you can do is put hygiene checks. Most reputed conferences wouldn't even do that. Also, how and why is this a feel-good policy? The program committee of the conference is reputed enough to check for fraud/discrepancies. After all, the Stapel fraud (like many others) was detected by his peers. FYI - this is not a journal either.
- jhdevos 14y ago
  You used the phrase 'to eliminate such frauds.' The suggestion (to me) is that this is a fool proof method that eliminates such frauds. I would contend this is not true; fraud is still relatively easy. The question is how much this promotes a false sense of security. Stapel's fraud was only detected after many years, and after his results had been widely published. I still hear some of his conclusions quoted now and then as if it were established truth, by people who have not yet realised that it is one of the famous Stapel ones.
  - denzil_correa 14y ago
    Some clarifications - + No method/suggestion is fool proof in the world of publications; simply can't be + Stapel's fraud was "data fraud" which is eliminated if your work on publicly available datasets + NLP community encourages work on public data and you can't get Stapel level of recognition if you do not have your data/tools out in the open + Any suggestion/method if not fool proof can be said to promote a false sense of security; it is important to start somewhere rather than maintain status quo