8 ms·

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I lo

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insincerely opensourced projects.

GuyPostington 7y ago
Have you been able to reach out to the author and get clarification?
- billconan 7y ago
  I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers. They didn't even bother to merge my pull requests.
  - scott_s 7y ago
    There's probably a disconnect in expectations. The researchers are probably publishing the code more as a document, to say, hey, this is how we did this. It sounds like you have expectations that the code is part of an ongoing project. But ongoing projects require maintenance, and the researchers have likely moved on to new things.
colechristensen 7y ago
It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.
- raverbashing 7y ago
  > Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) Absolutely correct, and not in a good way Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)
  - curl_e 7y ago
    The good news is people are trying to address this. Starting in the UK there is a movement for Research Software Engineering https://rse.ac.uk/ https://rse.ac.uk/ and happening right now is the German RSE conference https://www.de-rse.org/en/conf2019/ https://www.de-rse.org/en/conf2019/
- dTal 7y ago
  >It's just good news when you can find source code at all instead of just being told vague things about something the author did. This is extremely common, and I don't see why it should be allowed. I could easily write a paper claiming some kind of interesting result, fake a few "ground truth vs my algorithm" pairs, and talk a bit of mathy-sounding piffle about what it does - and it would be indistinguishable from many papers I have read. In my opinion - provide runnable source, or what you have isn't a paper, it's a boast.
- itronitron 7y ago
  In my opinion, a paper should describe the contribution in sufficient detail that it can be implemented by someone with similar expertise. I believe that would foster greater innovation as people would necessarily have greater familiarity with all of the details, and they would better see why the thing works or doesn't.
liuliu 7y ago
Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to < 20% top-1 error rate. That should be left to TensorFlow model zoo or GluonCV model zoo. I simply look at these research "open-source" as reproducible research.
- PaulHoule 7y ago
  Yeah but I wonder what fraction of CS papers test the algorithm that they thought they were testing...
  - seanmcdirmid 7y ago
    A lot of CS papers aren’t testing algorithms at all.
    - opportune 7y ago
      It is very common in the last few pages of a CS paper to benchmark your results against other algorithms or models
      - seanmcdirmid 7y ago
        And it’s also very common not to. CS papers are pretty diverse in what they are talking about, many aren’t algorithm papers.
- billconan 7y ago
  I'm not asking for much, but the code should be runnable at least. This is why I think jupyter notebook and things alike are so important as the future publishing media. Reproducibility is very important.
  - 7y ago
nickpsecurity 7y ago
One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper. This sad situation is both bad for open source and science. Bad for science since inaccurate results that probably aren't even reproducible aren't the knowledge advancements we need. Funding agencies like NSF need to change their policies to address the root cause here. Universities will follow where the incentives lead them.
- not2b 7y ago
  Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.
  - YeGoblynQueenne 7y ago
    In machine learning, you don't get credit for publishing rigorous papers. You get credits for publishing papers that show improved performance: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. Geoff Hinton interview: https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!". But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks. If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.
throwaway287391 7y ago
This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research, but I'll clean up all the code, rerun experiments to make sure it all still works like before, and add a bunch of documentation. But no, still not good enough -- it has to run at the press of a button in your particular programming environment. If we don't know how to write a script (or couldn't be bothered to spend the time writing one) to check that the data is on disk and, if not, crawl a website to download some huge dataset in one click, test our code on your OS, your CPU/GPU/TPU/..., etc., we were being "insincere" with our open-sourcing efforts.
- liquidify 7y ago
  If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.
  - draw_down 7y ago
    I'd say that depends on the readers an awful lot.
  - throwaway287391 7y ago
    What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.
nullc 7y ago
There appears to be something of a software engineering crisis in academia. Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuable enabling other people's work to be a principle researcher themselves. I've heard from a number of people that they avoid hiding their programming skills to avoid that trap. This has many consequences beyond the obvious direct ones like papers with unusable implementations. Academic work is often valuable even when it isn't practically useful but in many cases where academics are /trying/ to do something practical they fail because their community lacks the engineering experience-- I've seen a fair number of papers presenting optimizations as useful engineering when in reality their approach only makes sense because of inefficient non-programmer tools make a weird set of primitives fast (e.g. using a matrix multiply in matlab where in C you'd simply write a loop). As a practitioner this is frustrating because it sometimes requires re-implementing the approach to realize that it was only 'fast' compared to a pants-on-head-foolish approach. It also can create essentially fake results. Many times the lack of software engineering is compensated for by mocking components out in ways that would give faithful results if the researcher understood everything, but the whole point of expirementation is that the researcher doesn't! This mocking is seldom disclosed in papers. For example, I've encountered multiple papers that claimed to implement some enhancement to Bitcoin and tested it in a test bed but in reality they just added sleep()s with appropriate times, guessed out of specious reasoning like "our function X should be 10x slower than a signature", and hardcoded constants (computed in mathmatica or sage) for their messages. ... not realizing that sleep() isn't the best analog for cpu busywork in a multithreaded program! Another kind of fake result I've encountered which is even less directly a result of the software engineering shortage is in signal processing literature. While working on audio/video compression I found it common for algorithms to be presented without various constants and after reimplementing and asking the authors for their constants I found that they'd been cherrypicked for the ten images used in the paper, and that the whole approach doesn't actually work. This is a kind of ineptitude (or outright dishonesty) that would be much less common in a world where reviewers received a working and usable implementation in source form-- but that can't be expected in a world where qualified software engineering is not readily available to researchers. I don't have any proposed solutions but I think it's important to acknowledge that it is a common and serious limitation to the usefulness and accuracy of contemporary research.
krageon 7y ago
I had the great displeasure to work on multiple state-of-the-art algorithms related to sticking things in buckets with varying degrees of optimisation and/or supervision. There are a ton of reasonably recent papers for that, some of them have source code (most of them have nigh-impenetrable theory on why their algorithm is the best) and every case I tested was emphatically much worse than the technologies we've been using for 10+ years in any practical sense. That problem is compounded with the issues you describe, which I'd categorise as probably the best case. I did not find even a single case of a properly opensourced repository that just compiled somewhere that wasn't the researcher's laptop.
IfOnlyYouKnew 7y ago
https://github.com/jonadsimon/entendrepreneur-web https://github.com/jonadsimon/entendrepreneur-web?

GuyPostington 7y ago

Have you been able to reach out to the author and get clarification?

billconan 7y ago
I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers. They didn't even bother to merge my pull requests.
- scott_s 7y ago
  There's probably a disconnect in expectations. The researchers are probably publishing the code more as a document, to say, hey, this is how we did this. It sounds like you have expectations that the code is part of an ongoing project. But ongoing projects require maintenance, and the researchers have likely moved on to new things.

colechristensen 7y ago

It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.

raverbashing 7y ago
> Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) Absolutely correct, and not in a good way Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)
- curl_e 7y ago
  The good news is people are trying to address this. Starting in the UK there is a movement for Research Software Engineering https://rse.ac.uk/ https://rse.ac.uk/ and happening right now is the German RSE conference https://www.de-rse.org/en/conf2019/ https://www.de-rse.org/en/conf2019/
dTal 7y ago
>It's just good news when you can find source code at all instead of just being told vague things about something the author did. This is extremely common, and I don't see why it should be allowed. I could easily write a paper claiming some kind of interesting result, fake a few "ground truth vs my algorithm" pairs, and talk a bit of mathy-sounding piffle about what it does - and it would be indistinguishable from many papers I have read. In my opinion - provide runnable source, or what you have isn't a paper, it's a boast.
itronitron 7y ago
In my opinion, a paper should describe the contribution in sufficient detail that it can be implemented by someone with similar expertise. I believe that would foster greater innovation as people would necessarily have greater familiarity with all of the details, and they would better see why the thing works or doesn't.

liuliu 7y ago

Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to < 20% top-1 error rate. That should be left to TensorFlow model zoo or GluonCV model zoo. I simply look at these research "open-source" as reproducible research.

PaulHoule 7y ago
Yeah but I wonder what fraction of CS papers test the algorithm that they thought they were testing...
- seanmcdirmid 7y ago
  A lot of CS papers aren’t testing algorithms at all.
  - opportune 7y ago
    It is very common in the last few pages of a CS paper to benchmark your results against other algorithms or models
    - seanmcdirmid 7y ago
      And it’s also very common not to. CS papers are pretty diverse in what they are talking about, many aren’t algorithm papers.
billconan 7y ago
I'm not asking for much, but the code should be runnable at least. This is why I think jupyter notebook and things alike are so important as the future publishing media. Reproducibility is very important.
- 7y ago

nickpsecurity 7y ago

One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper. This sad situation is both bad for open source and science. Bad for science since inaccurate results that probably aren't even reproducible aren't the knowledge advancements we need. Funding agencies like NSF need to change their policies to address the root cause here. Universities will follow where the incentives lead them.

not2b 7y ago
Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.
- YeGoblynQueenne 7y ago
  In machine learning, you don't get credit for publishing rigorous papers. You get credits for publishing papers that show improved performance: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. Geoff Hinton interview: https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!". But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks. If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.

throwaway287391 7y ago

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research, but I'll clean up all the code, rerun experiments to make sure it all still works like before, and add a bunch of documentation. But no, still not good enough -- it has to run at the press of a button in your particular programming environment. If we don't know how to write a script (or couldn't be bothered to spend the time writing one) to check that the data is on disk and, if not, crawl a website to download some huge dataset in one click, test our code on your OS, your CPU/GPU/TPU/..., etc., we were being "insincere" with our open-sourcing efforts.

liquidify 7y ago
If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.
- draw_down 7y ago
  I'd say that depends on the readers an awful lot.
- throwaway287391 7y ago
  What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.

nullc 7y ago

There appears to be something of a software engineering crisis in academia. Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuable enabling other people's work to be a principle researcher themselves. I've heard from a number of people that they avoid hiding their programming skills to avoid that trap. This has many consequences beyond the obvious direct ones like papers with unusable implementations. Academic work is often valuable even when it isn't practically useful but in many cases where academics are /trying/ to do something practical they fail because their community lacks the engineering experience-- I've seen a fair number of papers presenting optimizations as useful engineering when in reality their approach only makes sense because of inefficient non-programmer tools make a weird set of primitives fast (e.g. using a matrix multiply in matlab where in C you'd simply write a loop). As a practitioner this is frustrating because it sometimes requires re-implementing the approach to realize that it was only 'fast' compared to a pants-on-head-foolish approach. It also can create essentially fake results. Many times the lack of software engineering is compensated for by mocking components out in ways that would give faithful results if the researcher understood everything, but the whole point of expirementation is that the researcher doesn't! This mocking is seldom disclosed in papers. For example, I've encountered multiple papers that claimed to implement some enhancement to Bitcoin and tested it in a test bed but in reality they just added sleep()s with appropriate times, guessed out of specious reasoning like "our function X should be 10x slower than a signature", and hardcoded constants (computed in mathmatica or sage) for their messages. ... not realizing that sleep() isn't the best analog for cpu busywork in a multithreaded program! Another kind of fake result I've encountered which is even less directly a result of the software engineering shortage is in signal processing literature. While working on audio/video compression I found it common for algorithms to be presented without various constants and after reimplementing and asking the authors for their constants I found that they'd been cherrypicked for the ten images used in the paper, and that the whole approach doesn't actually work. This is a kind of ineptitude (or outright dishonesty) that would be much less common in a world where reviewers received a working and usable implementation in source form-- but that can't be expected in a world where qualified software engineering is not readily available to researchers. I don't have any proposed solutions but I think it's important to acknowledge that it is a common and serious limitation to the usefulness and accuracy of contemporary research.

krageon 7y ago

I had the great displeasure to work on multiple state-of-the-art algorithms related to sticking things in buckets with varying degrees of optimisation and/or supervision. There are a ton of reasonably recent papers for that, some of them have source code (most of them have nigh-impenetrable theory on why their algorithm is the best) and every case I tested was emphatically much worse than the technologies we've been using for 10+ years in any practical sense. That problem is compounded with the issues you describe, which I'd categorise as probably the best case. I did not find even a single case of a properly opensourced repository that just compiled somewhere that wasn't the researcher's laptop.

IfOnlyYouKnew 7y ago

https://github.com/jonadsimon/entendrepreneur-web https://github.com/jonadsimon/entendrepreneur-web?