5 ms·

Yea, I have always found academic machine learning papers to have very little value. The helpful people create a github and write a blog post, not publish a pap

by nonbel 8y ago

Yea, I have always found academic machine learning papers to have very little value. The helpful people create a github and write a blog post, not publish a paper.

jackcosgrove 8y ago
Self-publishing code repositories or notebooks is clearly the superior approach, but unfortunately researchers are given less institutional credit if they are not published in an academic journal. The best approach is to get the benefits of peer review and then publish all your research artifacts alongside the paper. I am not sure if the academic journals prohibit this kind of thing though.
- PascLeRasc 8y ago
  I'm pretty certain PLOS allows and even encourages this: https://www.plos.org/open-access https://www.plos.org/open-access Elsevier is by far the worst publisher regarding open access though: https://twitter.com/protohedgehog/status/1028819653982736389?lang=en https://twitter.com/protohedgehog/status/1028819653982736389...
Voloskaya 8y ago
Because you are not the target audience for these papers. ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo. There are tens of interesting results that come out everyday. Papers with abstract and sections is the most efficient way to understand the overall idea. Blog post are nice for reaching outside of the research community. Code should always be linked for people that want to reproduce, but they don't replace a paper.
- nonbel 8y ago
  >"ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo." I don't see why this would take any more time than properly studying a paper, in fact it should be quicker since the information is presented in the proper format (not translated from code to math/prose).
  - Voloskaya 8y ago
    Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility. Also once you are experienced in the field, you don't relearn everything for every new paper. Most of the paper just propose incremental minor changes. The only things I care about is what these changes are (conceptually, not a code comparison) and data about the effect of these changes. I don't want to have to read your code to find out where you have applied a certain new hyper-parameter to your gradient descent. Stop thinking that the entire research population is publishing all results as papers because they don't know better. It's indeed the most efficient way to share knowledge for experts in a given field. > since the information is presented in the proper format (not translated from code to math/prose) Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations? You have got it completely backward.
    - nonbel 8y ago
      >"Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility." I have much experience reading research literature (but not ML specifically). The vast majority of the time I just skim a paper looking for certain info. What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper. >"Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations?" This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation.
      - temp_account00 8y ago
        Engineering researcher, but not-ML. "Code is everything" approach presumes that communication is computational by default. I'm not sure if researchers agree on that. This is particularly important for a field that aspire for artificial intelligence. Language is the best bet we have at the moment. Secondly, there are social aspects. I am becoming more well read in my field, and there are time when genuine "rediscovery" occurs. Many phd students, depending on their research group, do not come up with ground breaking work right off the bat. It takes them a few years. In the publish and perish economy, there are venues to show your paper. If their genuine work is rejected, it may stop them from progressing in their career to come up with great work. It is like expecting an undergrad to come up with a full master's thesis for a course project. Happens, but not often. Above being said, now if I am to read the code, I will be rereading many code repretitions every year. Whereas reading similar abstracts is less time consuming. Your blogpost comment touches a bigger issue of how to tell who conducts legitimate research. The best we have so far is that those at the top of their field to provide assessment. They're journal editors. They have dedicated their life to their field, and have read paprs from decades of research work (whether or not the field is scientifically paramount is irrelevant, not all can learn the same thing, and education system is there to _educate_ the population at large, along with generating new knowledge, and along with advancing new researchers). For the sake of completeness I'd add this, at least in my case and perhaps many others, as one becomes more experienced, one can assess their earlier grasp of the field, or their earlier misunderstandings, better. Hence it seems reasonable to publish, and have abstracts.

by nonbel 8y ago

Yea, I have always found academic machine learning papers to have very little value. The helpful people create a github and write a blog post, not publish a paper.

jackcosgrove 8y ago

Self-publishing code repositories or notebooks is clearly the superior approach, but unfortunately researchers are given less institutional credit if they are not published in an academic journal. The best approach is to get the benefits of peer review and then publish all your research artifacts alongside the paper. I am not sure if the academic journals prohibit this kind of thing though.

PascLeRasc 8y ago
I'm pretty certain PLOS allows and even encourages this: https://www.plos.org/open-access https://www.plos.org/open-access Elsevier is by far the worst publisher regarding open access though: https://twitter.com/protohedgehog/status/1028819653982736389?lang=en https://twitter.com/protohedgehog/status/1028819653982736389...

Voloskaya 8y ago

Because you are not the target audience for these papers. ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo. There are tens of interesting results that come out everyday. Papers with abstract and sections is the most efficient way to understand the overall idea. Blog post are nice for reaching outside of the research community. Code should always be linked for people that want to reproduce, but they don't replace a paper.

nonbel 8y ago
>"ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo." I don't see why this would take any more time than properly studying a paper, in fact it should be quicker since the information is presented in the proper format (not translated from code to math/prose).
- Voloskaya 8y ago
  Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility. Also once you are experienced in the field, you don't relearn everything for every new paper. Most of the paper just propose incremental minor changes. The only things I care about is what these changes are (conceptually, not a code comparison) and data about the effect of these changes. I don't want to have to read your code to find out where you have applied a certain new hyper-parameter to your gradient descent. Stop thinking that the entire research population is publishing all results as papers because they don't know better. It's indeed the most efficient way to share knowledge for experts in a given field. > since the information is presented in the proper format (not translated from code to math/prose) Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations? You have got it completely backward.
  - nonbel 8y ago
    >"Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility." I have much experience reading research literature (but not ML specifically). The vast majority of the time I just skim a paper looking for certain info. What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper. >"Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations?" This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation.
    - temp_account00 8y ago
      Engineering researcher, but not-ML. "Code is everything" approach presumes that communication is computational by default. I'm not sure if researchers agree on that. This is particularly important for a field that aspire for artificial intelligence. Language is the best bet we have at the moment. Secondly, there are social aspects. I am becoming more well read in my field, and there are time when genuine "rediscovery" occurs. Many phd students, depending on their research group, do not come up with ground breaking work right off the bat. It takes them a few years. In the publish and perish economy, there are venues to show your paper. If their genuine work is rejected, it may stop them from progressing in their career to come up with great work. It is like expecting an undergrad to come up with a full master's thesis for a course project. Happens, but not often. Above being said, now if I am to read the code, I will be rereading many code repretitions every year. Whereas reading similar abstracts is less time consuming. Your blogpost comment touches a bigger issue of how to tell who conducts legitimate research. The best we have so far is that those at the top of their field to provide assessment. They're journal editors. They have dedicated their life to their field, and have read paprs from decades of research work (whether or not the field is scientifically paramount is irrelevant, not all can learn the same thing, and education system is there to _educate_ the population at large, along with generating new knowledge, and along with advancing new researchers). For the sake of completeness I'd add this, at least in my case and perhaps many others, as one becomes more experienced, one can assess their earlier grasp of the field, or their earlier misunderstandings, better. Hence it seems reasonable to publish, and have abstracts.