6 ms·

Is there some background reason that academics generally don't provide code in their papers? Hell, in a lot of papers I've tried looking at recently[0], there w

• mayank 7y ago
  Research code is generally an abomination, scraped together by MS/PhD students who may never have had any industry exposure to best practices, norms, testing, etc. There may also be the fear of releasing buggy code, which if/when discovered, will discredit the paper. Far safer to just not release the code, since it isn't mandated (which should absolutely 100% change) Source: am former producer/publisher of abomination research code.
  • dv_dt 7y ago
    I wonder if a journal could be established that emphasizes code integrated papers, something that might accept submissions as jupyter notebooks for example. Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
    • scribu 7y ago
      Formatting articles as jupyter notebooks might help with reproducibility, but it wouldn't necessarily help with understanding. distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/ https://distill.pub/journal/ It's a lot of work, tho: > One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
      • dv_dt 7y ago
        Reproducibility is one of the core problems of the article. I'm not sure any format can assure understanding by itself - that's more the purview of the reviewers I would think. Applying the criticism to my own suggestion: even the reproducibility issue is only tangentially helped by a Jupyter format, it may only bring reviewers to the point where one might execute the notebook (though there are still issues there if its an exceeding long computation or needs special hardware setup, or the notebook executes w/o demonstrating actual computation).
    • tsumnia 7y ago
      This is a current area of research as the prevalence of interactive notebooks has peaked. As of now, they simply "EXIST" and we know we like them. Its still unclear as to why; obviously having code intertwined other common communication mediums but what about that makes us like it. What benefits does it have over pseudocode or just code? What about code beyond small scripts (500+ lines with design patterns or other abstractions)?
    • codebje 7y ago
      Jupyter notebooks are risky business for reproducible work. No dependency data means they're highly prone to bit rot. Stored results and out-of-order execution means they're prone to subtle errors. Environmental leakage is relatively high. Literate programming suffers from this in general because you don't want to clutter up your document with noise about versions and so forth - even as appendices, including package information and build instructions is a lot of noise. Requiring that papers making claims about some code base publish that code base simultaneously with the paper via the peer review process should be sufficient to improve the overall state of affairs. Don't underestimate the value of demos and competitions with companion papers either, those tend to get a lot more notice.
      • disgruntledphd2 7y ago
        I dunno, I get where you are coming from with respect to literate programming, but I find that it's often better to show all the versions in an org file (or whatever tool you use) and write up a report separately including the final results. In general, you'll have a lot of approaches that don't work out which are nice to have a record of, but definitely don't merit being in the final paper.
        • codebje 7y ago
          A published code base doesn't necessarily mean a full revision history, just something others can reproducibly build and run. If your claims don't depend on specific behaviours of some body of code, you wouldn't need it - eg, an article claiming some asymptotic performance of an algorithm should describe the algorithm in the abstract s.t. the performance bound can be proven, not some specific language's implementation of the algorithm.
      • dv_dt 7y ago
        Seems like one should be able to tag some metadata to a notebook - as little as a git url and a hash or as exotic as an IPFS link or (I hate to say it) some other blockchain info. Somehow dependencies would seem to be something needed to be specified in the submittal standards of a code-required journal
    • dilawar 7y ago
      There are some journals who have started taking it seriously. eLife does that: asks authors to put their code on GitHub/gitlab etc. Fork their repo and use that as snapshot for the published paper. I like the eLife approach. But for most journals, it's the story as usualy. Personally I dont feel very confident about a paper result if they don't publish the code. Why hold back on the code?
  • khawkins 7y ago
    I disagree that it is a lack of industry exposure. "Best practices, norms, testing, etc." is a waste of time when you develop well written code that doesn't produce good results. After months of tweaking even a good foundation of code, it begins looking atrocious. But you get graduated for publications based on good results, not for code that someone else can use.
    • kian 7y ago
      testing, refactoring, code organization, consistent code styling for easy search-replace, and other best practice norms will save you months of effort in tracking down bugs, fixing new ones you introduced while scrambling around researching, figuring out what's going on when you have an idea that uses code you wrote six months ago, etc. Sure, maybe a tutorial and documentation might not be worth the effort for something that hasn't produced results (does publishable research still count as not having produced results?), but most industry best practices I can think of will save you much more than the time invested even on relatively modest (< 1 month) projects.
    • majormajor 7y ago
      > After months of tweaking even a good foundation of code, it begins looking atrocious. Months of tweaking code is exactly what you want some test coverage for - so that you end up with a tweaked version of something where you still know what it does instead of something that does god knows what.
    • closeparen 7y ago
      Typing “git commit” when you reach a milestone costs nothing, and can save hours of trying to get back to a previous working state.
  • Reelin 7y ago
    > Research code is generally an abomination ... http://matt.might.net/articles/crapl http://matt.might.net/articles/crapl > The CRAPL is an open source "license" for academics that encourages code-sharing, regardless of how much how much Red Bull and coffee went into its production.
  • Someone 7y ago
    In addition, you may want to milk your code for more papers. Releasing the code makes it easier for others to also do that milking, too, finding and publishing some results before you can find them.
  • solveit 7y ago
    > There may also be the fear of releasing buggy code, which if/when discovered, will discredit the paper. I know you know this, but this is absolutely a feature.
  • hn23 7y ago
    Well, he was asking for pseudo code. So this could be at a high level where you do not really have to fear bugs, do you?
  • skunkpocalypse 7y ago
    Yeah the big question isn't why code isn't published. The big question is why this failure to publish code is tolerated by program+steering committees.
• ACow_Adonis 7y ago
  If you saw the code academics and students write, you'd know why they don't want to publish it. Also, more likely and non- nefarious, but no less harmful, there's just an academic culture that says "this is what a paper looks like and this is what it contains". Moving from this pattern increases the odds you'll not be published, and it would also likely result in a scandal when someone uses it to point out that the peers in peer review are neither peers (they can't read nor understand nor test the code) nor reviewing (they won't actually put resources into code review).
  • DennisP 7y ago
    Sorta makes you wonder whether this messy, poorly-tested, unreviewed code is correctly implementing the algorithms they think they're testing.
    • thom 7y ago
      In my experience, not reliably. And when following up something suspicious, you'll hear "I don't know, I didn't write that".
      • krick 7y ago
        That's only more the reason why they have to publish it.
    • ACow_Adonis 7y ago
      Oh I don't wonder :) After a few years experience with analysis you sometimes see certain patterns in papers which strongly suggest bugs you've experienced, phenomenon in the data set you're already familiar with, and hacking/overfitting. And some of the languages are specifically designed to keep running and produce wrong results rather than crash and abort: a cynical man might say this is a feature not a bug and that's why they're popular. As a non-wealthy, non-academic, this isn't a battle I'm interested in fighting directly. There are other hills I'd rather die on :)
• YeGoblynQueenne 7y ago
  Well. In theory, one should be able to reproduce (implement) an algorithm described in a paper just by reading the paper. It's better if the algorithm is typeset as, you know, an algorithm, but if the algorithm can be better described in plain English... Besides, _ideally_, machine learning reseach is not about building a system but about making a theoretical claim that is supported by empirical results. The main claim should be about a property of the algorithm: its time complexity, its convergence, its correctness, etc. Experiments should directly test the theoretical claim and verify it in practice. ... ideally. In practice most machine learning papers are vehicles for a little table that compares your system with a baseline that you chose based on the fact that your system beats it (or you wouldn't have put it in the paper). Whether your experiments prove anything beyond the fact that you beat some benchmark is not really important. And so whether someone publishes a reproducible paper is of no consequence because the only point of papers is to pass review and convince someone that the researchers should stay in the (research) money. In any case, things move so fast that most of the time nobody wants to reproduce your old and unsexy paper from last week. Then some busybody comes along and points out the fact that you're wearing no clothes. Like in the paper above.
• krick 7y ago
  Once again, I'll attribute to malice what many here are so eager to attribute to stupidity (or something rather similar). Basically, they don't have any incentive to do so. The goal of the researcher is to be marked as an author of something that will be considered as an important result in the domain, not to actually help the others to understand the result and to be able to build upon it. The paragraphs of English text help them to be able to claim that they are indeed the authors of some novel method (exactly what the author wants). Code/data accessibility helps others to both find errors/imperfections in their approach and to build upon their results (exactly what the author doesn't want).
• vidarh 7y ago
  I did my thesis on OCR, and even simple binary thresholding algorithms (to decide a cutoff point for "background" vs "text") that might boil down to a dozen lines of code if that in some cases, would often be presented as a formula (great; at least something) or imprecise prose, but where you then when translating it into code would find that their results depended on setting certain parameters to certain values that they'd not specified anywhere. In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range. It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data. I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway. I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.