3 ms·

>"ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo." I don't see why this wo

by nonbel 8y ago

>"ML researchers (or any researcher for that matter) don't have time to go through a long form blogpost or skim through a GitHub repo." I don't see why this would take any more time than properly studying a paper, in fact it should be quicker since the information is presented in the proper format (not translated from code to math/prose).

Voloskaya 8y ago
Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility. Also once you are experienced in the field, you don't relearn everything for every new paper. Most of the paper just propose incremental minor changes. The only things I care about is what these changes are (conceptually, not a code comparison) and data about the effect of these changes. I don't want to have to read your code to find out where you have applied a certain new hyper-parameter to your gradient descent. Stop thinking that the entire research population is publishing all results as papers because they don't know better. It's indeed the most efficient way to share knowledge for experts in a given field. > since the information is presented in the proper format (not translated from code to math/prose) Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations? You have got it completely backward.
- nonbel 8y ago
  >"Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility." I have much experience reading research literature (but not ML specifically). The vast majority of the time I just skim a paper looking for certain info. What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper. >"Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations?" This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation.
  - temp_account00 8y ago
    Engineering researcher, but not-ML. "Code is everything" approach presumes that communication is computational by default. I'm not sure if researchers agree on that. This is particularly important for a field that aspire for artificial intelligence. Language is the best bet we have at the moment. Secondly, there are social aspects. I am becoming more well read in my field, and there are time when genuine "rediscovery" occurs. Many phd students, depending on their research group, do not come up with ground breaking work right off the bat. It takes them a few years. In the publish and perish economy, there are venues to show your paper. If their genuine work is rejected, it may stop them from progressing in their career to come up with great work. It is like expecting an undergrad to come up with a full master's thesis for a course project. Happens, but not often. Above being said, now if I am to read the code, I will be rereading many code repretitions every year. Whereas reading similar abstracts is less time consuming. Your blogpost comment touches a bigger issue of how to tell who conducts legitimate research. The best we have so far is that those at the top of their field to provide assessment. They're journal editors. They have dedicated their life to their field, and have read paprs from decades of research work (whether or not the field is scientifically paramount is irrelevant, not all can learn the same thing, and education system is there to _educate_ the population at large, along with generating new knowledge, and along with advancing new researchers). For the sake of completeness I'd add this, at least in my case and perhaps many others, as one becomes more experienced, one can assess their earlier grasp of the field, or their earlier misunderstandings, better. Hence it seems reasonable to publish, and have abstracts.
  - Voloskaya 8y ago
    > What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper. The whole point of a blog post is vulgarization (which is definitely needed as well): explaining an idea without assuming that the reader has a lot of prior knowledge in the domain. Which goes in the opposite direction of what researchers look for: the most efficient way to understand what is being proposed. Most of ML papers abstract would be pretty horrible on a blog post in term of vulgarization. Plus, papers are all very well centralized in Arxiv. How am I supposed to find blog posts? I should just hope that what I am interested in will blowup on twitter or /r/machinelearning and I will notice it? This sounds extremely sub-optimal. > This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation. Do you also think that biologist should stop writing papers and just upload video of their actual experiments? I work in an ML research lab with 30 or so researchers, research never begins with some code snippets. It always begins on the whiteboard. Even between us, where we have the code and data readily available to share with each others, when someone presents it's work, they will never show code, but always equations or model diagrams. It's just an order of magnitude easier to visualize. Take this paper for example: https://arxiv.org/pdf/1805.11604v3.pdf https://arxiv.org/pdf/1805.11604v3.pdf how would you even begin to communicate something like that through code?

by nonbel 8y ago

Voloskaya 8y ago

Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility. Also once you are experienced in the field, you don't relearn everything for every new paper. Most of the paper just propose incremental minor changes. The only things I care about is what these changes are (conceptually, not a code comparison) and data about the effect of these changes. I don't want to have to read your code to find out where you have applied a certain new hyper-parameter to your gradient descent. Stop thinking that the entire research population is publishing all results as papers because they don't know better. It's indeed the most efficient way to share knowledge for experts in a given field. > since the information is presented in the proper format (not translated from code to math/prose) Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations? You have got it completely backward.

nonbel 8y ago
>"Because you don't "properly" study all papers that you read. For most, you just want to know the general ideas and understand the novel techniques that were applied. You might decide to do a deep dive in a very select few. At this point code becomes useful for reproducibility." I have much experience reading research literature (but not ML specifically). The vast majority of the time I just skim a paper looking for certain info. What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper. >"Do you really think researchers just write some random code until they come up with good results and then try to retrofit that with some equations?" This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation.
- temp_account00 8y ago
  Engineering researcher, but not-ML. "Code is everything" approach presumes that communication is computational by default. I'm not sure if researchers agree on that. This is particularly important for a field that aspire for artificial intelligence. Language is the best bet we have at the moment. Secondly, there are social aspects. I am becoming more well read in my field, and there are time when genuine "rediscovery" occurs. Many phd students, depending on their research group, do not come up with ground breaking work right off the bat. It takes them a few years. In the publish and perish economy, there are venues to show your paper. If their genuine work is rejected, it may stop them from progressing in their career to come up with great work. It is like expecting an undergrad to come up with a full master's thesis for a course project. Happens, but not often. Above being said, now if I am to read the code, I will be rereading many code repretitions every year. Whereas reading similar abstracts is less time consuming. Your blogpost comment touches a bigger issue of how to tell who conducts legitimate research. The best we have so far is that those at the top of their field to provide assessment. They're journal editors. They have dedicated their life to their field, and have read paprs from decades of research work (whether or not the field is scientifically paramount is irrelevant, not all can learn the same thing, and education system is there to _educate_ the population at large, along with generating new knowledge, and along with advancing new researchers). For the sake of completeness I'd add this, at least in my case and perhaps many others, as one becomes more experienced, one can assess their earlier grasp of the field, or their earlier misunderstandings, better. Hence it seems reasonable to publish, and have abstracts.
- Voloskaya 8y ago
  > What I am reading from you is that basically the only useful part of the ML paper is the abstract. If the repo/blog had a good abstract there would be no point to the paper. The whole point of a blog post is vulgarization (which is definitely needed as well): explaining an idea without assuming that the reader has a lot of prior knowledge in the domain. Which goes in the opposite direction of what researchers look for: the most efficient way to understand what is being proposed. Most of ML papers abstract would be pretty horrible on a blog post in term of vulgarization. Plus, papers are all very well centralized in Arxiv. How am I supposed to find blog posts? I should just hope that what I am interested in will blowup on twitter or /r/machinelearning and I will notice it? This sounds extremely sub-optimal. > This sounds like a strawman, but yea I do think the code is the "actual" method while the math is some idealized cleaned-up translation. Do you also think that biologist should stop writing papers and just upload video of their actual experiments? I work in an ML research lab with 30 or so researchers, research never begins with some code snippets. It always begins on the whiteboard. Even between us, where we have the code and data readily available to share with each others, when someone presents it's work, they will never show code, but always equations or model diagrams. It's just an order of magnitude easier to visualize. Take this paper for example: https://arxiv.org/pdf/1805.11604v3.pdf https://arxiv.org/pdf/1805.11604v3.pdf how would you even begin to communicate something like that through code?