4 ms·

You should always want to have the underlying code available. Without the exact procedures they used to process their data, the only kind of "using their concl

by bloaf 5y ago

You should always want to have the underlying code available. Without the exact procedures they used to process their data, the only kind of "using their conclusions" you can do is the superficial "take it at face value" kind. So many important details get hand-waved away in papers that say things like "we used the well known blahblahblah method to analyze the data." If you do it right, the code should in no way interfere with your ability to read abstracts.

deleted 5y ago
[deleted]
eesmith 5y ago
I think I can convince you otherwise. If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers listed at https://en.wikipedia.org/wiki/RSA_Factoring_Challenge https://en.wikipedia.org/wiki/RSA_Factoring_Challenge , then I think people will take it seriously, and not consider it at the superficial level. Even if I don't publish the algorithm. ("Because of the security implications of this work, I have decided to withhold publication for a year.") Furthermore, some things are worth publishing even if the methods was "it came to me in a dream" à la Kekulé's snake. If you can demonstrate a sorting network of size 47 for n=14 input (which is the known lowest bound) then you can publish that exemplar, even without publishing the method used to generate it. (If you used computer assistance then that method would likely also be publishable, but that's a different point. Newton famously used the calculus to solve problems, but published their proofs using more traditional approaches.) If you can come up with a protein model that is a significantly better fit to the X-ray diffraction data, then that's publishable too, no matter how you came up with that model. In all of these cases, there are ways to verify the validity of the results without reproducing the methods used to come up with the result.
- deleted 5y ago
  [deleted]
- lonesword 5y ago
  This won’t work for empirical research. I vividly recall weeks spent trying to reproduce a paper on information retrieval (a deep learning model). What saved me is skimming through the author’s codebase and chancing upon an undocumented sampling step. They were only using the first and last passage in a document as training data and uniformly sampling from 10% of the remaining passages, and the paper didn't mention this. I adopted their sampling strategy, and i was able to obtain their results. My argument is that there are nuances and subtleties that are often omitted in a paper (accidentally or otherwise), but are nevertheless required to reproduce the research.
  - eesmith 5y ago
    My example of a protein model is an example of empirical research, yes? My understanding is the X-ray gives you a diffraction pattern which is hard to invert to a structure, while if you have the structure the diffraction pattern is easy to compute. The diffraction pattern therefore gives you a way to verify that one model is a better fit than another model. It may not be perfect, certainly not. It might not even be correct once more data arrives. But if you predict a novel fold, and that fold matches the diffraction pattern significantly better than the current model, then it doesn't matter how you came up with the new fold, does it? It could have been a dream. It could have been search software. The result is still publishable. All of what you have said is true, but my point is for some research being able to verify the correctness of the result is all that matters, not being able to reproduce the research. Can you reproduce Kekulé's dream?
    - tsbischof 5y ago
      What do you see as the fundamental point of scientific communication? In your counterpoints you narrow in on papers being a means of communicating concepts or proof of work. In this view, showing the process itself is pointless or at least irrelevant to the main axiom. However, others (myself included) see the the communication of methods as a primary function of the literature, because this is what enables others to understand, critique, and build upon the idea.
      - 5y ago
- kragen 5y ago
  Yes, there are occasional exceptions where you don't have to repeat or replicate the experiments reported in a paper to verify them. But that is very much the exception. Generally you are expected to explain what you did in enough detail that the reader can replicate your experiment. If you're fitting a protein model to X-ray diffraction data, you aren't expected to include all the other protein models you considered that didn't fit, or explain to the reader your procedure for generating protein models, but you are expected to explain how you measured the fit to the X-ray diffraction data (with what algorithms or software, etc.) so that the reader can in theory do the same thing themself.
  - thfuran 5y ago
    >Yes, there are occasional exceptions where you don't have to repeat or replicate the experiments reported in a paper to verify them. But that is very much the exception. And even in this exceptional case, the algorithm itself is interesting above and beyond the fact of its existence.
    - kragen 5y ago
      It is, but if the algorithm produces a result such as a protein structure or a sorting network that is itself novel and verifiable, you can very reasonably publish that result separately. As long as it doesn't require knowing the search algorithm to replicate your result that the sorting network sorts correctly, which it wouldn't.
  - eesmith 5y ago
    Sure, but "I found the structure after 5 months playing around with it in Foldit" isn't that reproducible or informative either. The result is still the same - a novel fold which is a significantly better fit than existing modules, based on measured vs. predicted x-ray diffraction patterns and whatever other data you might have. Which is publishable, yes? When the Wikipedia entry at https://en.wikipedia.org/wiki/Foldit https://en.wikipedia.org/wiki/Foldit says "Foldit players reengineered the enzyme by adding 13 amino acids, increasing its activity by more than 18 times", how is that much different than "A magical wizard added 13 amino acids, increasing its activity by more than 18 times"? Or "secret software". What's publishable is that the result is novel (and hopefully interesting), and can be verified. The publication does not require that all step can be repeated.
- necovek 5y ago
  > If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers If some factors of those numbers are also large composites, without access to a good algorithm, nobody can truly verify your claims. If not and you include all of those factors in an easily digestible way for computers to process (let's call that "code"), it will be easy for anyone to reproduce your results (run that code which multiplies all the factors and gets the resulting RSA numbers). With code, they could easily check that there's not an error in your verification method too (eg. large number multiplication broken). This would achieve both goals: you'd withhold your algorithm for security reasons, and your results would be easier to verify. Edit: but to be honest, I think withholding the research is a bit of a special case. You are doing it on purpose, and you can easily offer a service to prove your algorithm works (eg. imagine a "factoring" web service that instantly gives you a hash of the resulting sequence of factors, and then only mails you the actual sequence in two days).
  - aliceryhl 5y ago
    It's a lot easier to check if something is a prime than to factor it.

by bloaf 5y ago

deleted 5y ago

[deleted]

eesmith 5y ago

I think I can convince you otherwise. If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers listed at https://en.wikipedia.org/wiki/RSA_Factoring_Challenge https://en.wikipedia.org/wiki/RSA_Factoring_Challenge , then I think people will take it seriously, and not consider it at the superficial level. Even if I don't publish the algorithm. ("Because of the security implications of this work, I have decided to withhold publication for a year.") Furthermore, some things are worth publishing even if the methods was "it came to me in a dream" à la Kekulé's snake. If you can demonstrate a sorting network of size 47 for n=14 input (which is the known lowest bound) then you can publish that exemplar, even without publishing the method used to generate it. (If you used computer assistance then that method would likely also be publishable, but that's a different point. Newton famously used the calculus to solve problems, but published their proofs using more traditional approaches.) If you can come up with a protein model that is a significantly better fit to the X-ray diffraction data, then that's publishable too, no matter how you came up with that model. In all of these cases, there are ways to verify the validity of the results without reproducing the methods used to come up with the result.

deleted 5y ago
[deleted]
lonesword 5y ago
This won’t work for empirical research. I vividly recall weeks spent trying to reproduce a paper on information retrieval (a deep learning model). What saved me is skimming through the author’s codebase and chancing upon an undocumented sampling step. They were only using the first and last passage in a document as training data and uniformly sampling from 10% of the remaining passages, and the paper didn't mention this. I adopted their sampling strategy, and i was able to obtain their results. My argument is that there are nuances and subtleties that are often omitted in a paper (accidentally or otherwise), but are nevertheless required to reproduce the research.
- eesmith 5y ago
  My example of a protein model is an example of empirical research, yes? My understanding is the X-ray gives you a diffraction pattern which is hard to invert to a structure, while if you have the structure the diffraction pattern is easy to compute. The diffraction pattern therefore gives you a way to verify that one model is a better fit than another model. It may not be perfect, certainly not. It might not even be correct once more data arrives. But if you predict a novel fold, and that fold matches the diffraction pattern significantly better than the current model, then it doesn't matter how you came up with the new fold, does it? It could have been a dream. It could have been search software. The result is still publishable. All of what you have said is true, but my point is for some research being able to verify the correctness of the result is all that matters, not being able to reproduce the research. Can you reproduce Kekulé's dream?
  - tsbischof 5y ago
    What do you see as the fundamental point of scientific communication? In your counterpoints you narrow in on papers being a means of communicating concepts or proof of work. In this view, showing the process itself is pointless or at least irrelevant to the main axiom. However, others (myself included) see the the communication of methods as a primary function of the literature, because this is what enables others to understand, critique, and build upon the idea.
    - 5y ago
kragen 5y ago
Yes, there are occasional exceptions where you don't have to repeat or replicate the experiments reported in a paper to verify them. But that is very much the exception. Generally you are expected to explain what you did in enough detail that the reader can replicate your experiment. If you're fitting a protein model to X-ray diffraction data, you aren't expected to include all the other protein models you considered that didn't fit, or explain to the reader your procedure for generating protein models, but you are expected to explain how you measured the fit to the X-ray diffraction data (with what algorithms or software, etc.) so that the reader can in theory do the same thing themself.
- thfuran 5y ago
  >Yes, there are occasional exceptions where you don't have to repeat or replicate the experiments reported in a paper to verify them. But that is very much the exception. And even in this exceptional case, the algorithm itself is interesting above and beyond the fact of its existence.
  - kragen 5y ago
    It is, but if the algorithm produces a result such as a protein structure or a sorting network that is itself novel and verifiable, you can very reasonably publish that result separately. As long as it doesn't require knowing the search algorithm to replicate your result that the sorting network sorts correctly, which it wouldn't.
- eesmith 5y ago
  Sure, but "I found the structure after 5 months playing around with it in Foldit" isn't that reproducible or informative either. The result is still the same - a novel fold which is a significantly better fit than existing modules, based on measured vs. predicted x-ray diffraction patterns and whatever other data you might have. Which is publishable, yes? When the Wikipedia entry at https://en.wikipedia.org/wiki/Foldit https://en.wikipedia.org/wiki/Foldit says "Foldit players reengineered the enzyme by adding 13 amino acids, increasing its activity by more than 18 times", how is that much different than "A magical wizard added 13 amino acids, increasing its activity by more than 18 times"? Or "secret software". What's publishable is that the result is novel (and hopefully interesting), and can be verified. The publication does not require that all step can be repeated.
necovek 5y ago
> If I publish a paper saying I have an algorithm which can factor large composites, and in the paper publish the factors to all of the RSA numbers If some factors of those numbers are also large composites, without access to a good algorithm, nobody can truly verify your claims. If not and you include all of those factors in an easily digestible way for computers to process (let's call that "code"), it will be easy for anyone to reproduce your results (run that code which multiplies all the factors and gets the resulting RSA numbers). With code, they could easily check that there's not an error in your verification method too (eg. large number multiplication broken). This would achieve both goals: you'd withhold your algorithm for security reasons, and your results would be easier to verify. Edit: but to be honest, I think withholding the research is a bit of a special case. You are doing it on purpose, and you can easily offer a service to prove your algorithm works (eg. imagine a "factoring" web service that instantly gives you a hash of the resulting sequence of factors, and then only mails you the actual sequence in two days).
- aliceryhl 5y ago
  It's a lot easier to check if something is a prime than to factor it.