5 ms·

Expecting fully executable code to accompany every publication is kind of unique to the modern ML research Scene. As someone from a very different computational

Expecting fully executable code to accompany every publication is kind of unique to the modern ML research Scene. As someone from a very different computational research field, where zero code is the norm, not the exception, this reads as a somewhat entitled rant. Reimplementation of a paper is actually a test of the robustness of the results. If you download the code of a previous paper, there may be some assumptions hidden in the implementation that aren't obvious. So I would argue that simply downloading and re-executing the author's implementation does not constitute reproducible research. I know it is costly, but for actual reproduction, reimplementation is needed.

sillysaurusx 5y ago
I wasn't sure whether to post my edit as a separate comment or not, but I significantly expanded my comment just now, that helps explain my position. I'd be very interested in your thoughts on that position, because if it's mistaken, I shouldn't be saying it. It represents whatever small contribution I can make to fellow new ML researchers, which is roughly: "watch out." In short, for two years, I kept trying to implement stated claims -- to reproduce them in exactly the way you say here -- and they simply didn't work as stated. It might sound confusing that the claims were "simply wrong" or "didn't work." But every time I tried, achieving anything remotely close to "success" was the exception, not the norm. And I don't think it was because I failed to implement what they were saying in the paper. I agree that that's the most likely thing. But I was careful. It's very easy to make mistakes, and I tried to make none, as both someone with over a decade of experience (https://shawnpresser.blogspot.com/ https://shawnpresser.blogspot.com/) and someone who cares deeply about the things I'm talking about here. It takes hard work to reproduce the technique the way you're saying. I put all my heart and soul into trying to. And I kept getting dismayed, because people kept trying to convince me of things that either I couldn't verify (because verification is extremely hard, as you well know) or were simply wrong. So if I sound entitled, I agree. When I got into this job, as an ML researcher, I thought I was entitled to the scientific method. Or anything vaguely resembling "careful, distilled, correct knowledge that I can build on."
- msapaydin 5y ago
  I think that not being able to reproduce the results claimed in a paper is not specific to ML research. While working as a post-doc at a top university research lab, i spent years trying to understand how it can be that some software that was supposed to corresponds to the well cited paper did not even come close to reproducing the results of the said paper, and that the primary author went on to become a prof at a top university in the US. In short, scientific fraud is also quite common, in most academic papers.
  - sillysaurusx 5y ago
    Thank you!! i spent years trying to understand how it can be that some software that was supposed to corresponds to the well cited paper did not even come close to reproducing the results of the said paper, This was my exact experience. I didn’t understand why I kept having it, and kept blaming myself for not being careful enough. My code must be wrong, or the data, or something. Nah. It was the idea. Kept feeling like a kick in the gut, until here we are today, when I’m warning everyone that Karras, of all people, might publish such a thing. I really appreciate that you posted this, because I’m so happy I wasn’t alone in the feeling of “what’s going on, here…?”
    - fossuser 5y ago
      That seems like a worthwhile thing to publicize in and of itself? The replication crisis in psychology threw out 50% or so of supposed scientific results. If this (or just straight fraud) is common elsewhere, it seems like knowing about that would be a good thing for science.
      - msapaydin 5y ago
        I agree, most top conferences nowadays publish reviews openly, and address this issue I think. Also it is easier said than done, this is so endemic in so many different academic settings, not just in the US, but also in Europe.
- 5y ago
xksteven 5y ago
Sometimes reimplantation is impossible without the code and the paper goes on to win awards because it's by a famous scientist. Then if the reimplantation doesn't work most of the time the graduate students are blamed instead of the original work. There are always assumptions. At least with public code and models those assumptions are laid bare for all to see and potentially expose any bad assumptions.
nerdponx 5y ago
It's not an entitled rant, other fields just have dismally low standards.
- phreeza 5y ago
  I tried to give reasons why publishing code may actually not be universally desirable.
xwolfi 5y ago
Yeah but look at the first comment which said the opposite of what you say is normal. The first comment was praising the fact the code was published, the second that no it's not, followed by an autistic rant. To his defense, the spirit of his rant was valid, the letter made it sound entitled.
areeh 5y ago
I think the discrepancy between what actually happens in the code and the main ideas in the paper is a great point, and it touches on the parent commenter's goal of attaining knowledge. For instance, even if the results are reproducible, maybe the proposed key idea in the paper is not the piece of code with the largest impact on performance. Though, with the code available you might be able to discover exactly what the discrepancy is if there is one and you might be stuck at "it doesn't reproduce" otherwise

sillysaurusx 5y ago

I wasn't sure whether to post my edit as a separate comment or not, but I significantly expanded my comment just now, that helps explain my position. I'd be very interested in your thoughts on that position, because if it's mistaken, I shouldn't be saying it. It represents whatever small contribution I can make to fellow new ML researchers, which is roughly: "watch out." In short, for two years, I kept trying to implement stated claims -- to reproduce them in exactly the way you say here -- and they simply didn't work as stated. It might sound confusing that the claims were "simply wrong" or "didn't work." But every time I tried, achieving anything remotely close to "success" was the exception, not the norm. And I don't think it was because I failed to implement what they were saying in the paper. I agree that that's the most likely thing. But I was careful. It's very easy to make mistakes, and I tried to make none, as both someone with over a decade of experience (https://shawnpresser.blogspot.com/ https://shawnpresser.blogspot.com/) and someone who cares deeply about the things I'm talking about here. It takes hard work to reproduce the technique the way you're saying. I put all my heart and soul into trying to. And I kept getting dismayed, because people kept trying to convince me of things that either I couldn't verify (because verification is extremely hard, as you well know) or were simply wrong. So if I sound entitled, I agree. When I got into this job, as an ML researcher, I thought I was entitled to the scientific method. Or anything vaguely resembling "careful, distilled, correct knowledge that I can build on."

msapaydin 5y ago
I think that not being able to reproduce the results claimed in a paper is not specific to ML research. While working as a post-doc at a top university research lab, i spent years trying to understand how it can be that some software that was supposed to corresponds to the well cited paper did not even come close to reproducing the results of the said paper, and that the primary author went on to become a prof at a top university in the US. In short, scientific fraud is also quite common, in most academic papers.
- sillysaurusx 5y ago
  Thank you!! i spent years trying to understand how it can be that some software that was supposed to corresponds to the well cited paper did not even come close to reproducing the results of the said paper, This was my exact experience. I didn’t understand why I kept having it, and kept blaming myself for not being careful enough. My code must be wrong, or the data, or something. Nah. It was the idea. Kept feeling like a kick in the gut, until here we are today, when I’m warning everyone that Karras, of all people, might publish such a thing. I really appreciate that you posted this, because I’m so happy I wasn’t alone in the feeling of “what’s going on, here…?”
  - fossuser 5y ago
    That seems like a worthwhile thing to publicize in and of itself? The replication crisis in psychology threw out 50% or so of supposed scientific results. If this (or just straight fraud) is common elsewhere, it seems like knowing about that would be a good thing for science.
    - msapaydin 5y ago
      I agree, most top conferences nowadays publish reviews openly, and address this issue I think. Also it is easier said than done, this is so endemic in so many different academic settings, not just in the US, but also in Europe.
5y ago

xksteven 5y ago

Sometimes reimplantation is impossible without the code and the paper goes on to win awards because it's by a famous scientist. Then if the reimplantation doesn't work most of the time the graduate students are blamed instead of the original work. There are always assumptions. At least with public code and models those assumptions are laid bare for all to see and potentially expose any bad assumptions.

nerdponx 5y ago

It's not an entitled rant, other fields just have dismally low standards.

phreeza 5y ago
I tried to give reasons why publishing code may actually not be universally desirable.

xwolfi 5y ago

Yeah but look at the first comment which said the opposite of what you say is normal. The first comment was praising the fact the code was published, the second that no it's not, followed by an autistic rant. To his defense, the spirit of his rant was valid, the letter made it sound entitled.

areeh 5y ago

I think the discrepancy between what actually happens in the code and the main ideas in the paper is a great point, and it touches on the parent commenter's goal of attaining knowledge. For instance, even if the results are reproducible, maybe the proposed key idea in the paper is not the piece of code with the largest impact on performance. Though, with the code available you might be able to discover exactly what the discrepancy is if there is one and you might be stuck at "it doesn't reproduce" otherwise