8 ms·

It has always been like this in DL research. Also, providing justification doesn't necessarily help, because they are usually just guesses. The problem goes dee

It has always been like this in DL research. Also, providing justification doesn't necessarily help, because they are usually just guesses. The problem goes deeper than this. It's easy, and common, to make up justifications after the fact. You can try a bunch of random stuff, some thing randomly works, and then you make up an explanation for why it works that sounds plausible, but in reality is nothing more than a fictional narrative to convince reviewers (and possibly yourself). The underlying problem here are the incentives of the academic system. Reviewers reward good narratives and stories. Also, in DL especially, it's quite difficult to perform ablation studies or get statistical significance numbers due experiments taking a long time to run. I hate to say it, because that's not how science should work, but with the flood of papers, looking at who wrote a paper is probably one of the more reliable indicators of quality. The reason is that people with a reputation are putting their reputation at stake when publishing something. Publishing BS would reflect badly on them, so they tend to go through more quality control. An unknown researcher at an unknown lab has nothing to lose by flooding arxiv with BS papers, which on the surface are indistinguishable from high quality work.

jprafael 3y ago
> I hate to say it, because that's not how science should work I have the opposite view. Science should be incremental and authors should be incentivized to share their (interesting) findings early and often. This makes the community as whole move faster because you get more visibility, funding, man-hours dedicated to things that are on the leading edge of research. Consider a scenario where a researcher is required to explain exactly why some phenomenon happens. Maybe it took 1 year to find the original phenomenon and then it takes 10 years to explain it to a reasonable level. Everyone only gets the benefit of this research 11 years after. Now consider the opposite scenario. After 1 year the author publishes and gets the attention of fellow colleagues. Some of them will collaborate together adding more man-hours / year, reducing the total time. Some of them might have already discovered something similar and thus avoid all repeated work. Some of them might be better positioned to solve the explanation piece based on their field of expertise, personal interests, availability. All of this makes the innovation happen faster. What you often see (or should see in high quality papers) is an hypothesis of why something happens. This in itself is valuable. Many hypotheses are unproven until today. If you assume that these hypotheses are true you'll often find better results or find them faster; and if you don't you have discovered something interesting to report on.
- loveparade 3y ago
  I don't think we disagree. All I am saying is that if your goal is to filter papers by their likelihood of having a useful result, the best signal is the author and their reputation, not necessarily the content (unless there is perhaps some obviously amazing demo). We don't disagree that for the community as a whole the system of publishing early and often is much better than the alternative of imposing restrictions.
- tpoacher 3y ago
  This guys agree with you: https://www.octopus.ac/ https://www.octopus.ac/ Alas, whether their idea will withstand the current incentives structure in academia, remains to be seen.
- psyklic 3y ago
  I'm of the opposite view - publish only if you can prove it. If you reward half results, the field will be deluged with half truths. If someone publishes a hypothesis you're working on with only a half proof (and stands to gain the future credit), then there is little incentive to continue doing the work to prove it. This was actually a major issue one of my PhD advisors had, since it led to poor foundations for the field with little incentive to ensure their validity.
  - rdedev 3y ago
    We need something in between. The paper author may not know why something is happening but has showed that it is statistically significant. Maybe he just does not have the context or background but someone else along the way. Of course the assumption is that people would do replication studies but no one is incentivized to do them so better to be on the safer side
- low_tech_love 3y ago
  That works well only if people have no motivation to fabricate or embellish results, and/or if they fully share the code and experimental setup. But in general I agree.
gumby 3y ago
> Also, in DL especially, it's quite difficult to perform ablation studies or get statistical significance numbers due experiments taking a long time to run. Aww, the poor babies. This the the nature of working with complex systems (life sciences, physics, social sciences, etc): teasing out signal from very noisy data that is quite expensive (in time at least) to obtain. On the engineering side things can (but don't always) move faster with a different kind of rigour. And that applies to observational reports in the sciences as well. All of which is to say that there's nothing wrong with the flood of papers that have a bunch of colored boxes and some running code. But consider those to be the equivalent of most of the reports in the 18th century Royal Society: "I have just returned from being a passenger on the HMS Foobar; while on the Malabar coast I saw an odd insect that looked like this drawing" And conversely spend referees' precious time on actual science.
- godelski 3y ago
  I mean I'm with you on the side of "ML needs more math" because we're about as sciencey as psychology, but I'm not quite on the "aww poor babies" side (yet). It's just the nature of how sciences evolve. The truth is that with deep rigorous theory, things like physics are easier. I'd say the same for psych too. It's a crazy amount of stats you need to do to account for all the unknowns and uncertainties but I'm not sure why these sciences push the theory people away when they can't even be bothered to add an epsilon to their models (which are always linear), but I digress. The thing that concerns me more though is the push against math. We see it a lot in ML but weirdly enough reviews will reject you if you don't have equations so you get tons of papers just copy pasting math that adds nothing to the paper. It's really weird and a clear indication of the laziness of reviews and area chairs. What I tell my students is "you don't need math to make good models but you need math to know why your models are wrong." This seems to be becoming more important than ever. Hell, how many people know about the Whitney embedding theorem? That itself helps answer half my undergrad questions and I think it's relatively unknown. But you need both. The thinkers and tinkerers. Referring to my longer rant, industry has captured the research field. I'm really happy they're involved but we're responding in insane ways. Let academia be the thinkers and let them make small models and don't expect a $100M budget for every paper. It's the railroading that annoys me and I think it's anti science. But no one wants to even acknowledge it's existence. I'll go ahead and say there's not much use to GPT papers unless you work for OpenAI (not to be confused with LLM papers). Let people explore new architectures and don't make them compete with sota models that have billions of dollars and man hours put into them. Hype is killing our field and it's hard to see because we're a high functioning addict.

jprafael 3y ago

> I hate to say it, because that's not how science should work I have the opposite view. Science should be incremental and authors should be incentivized to share their (interesting) findings early and often. This makes the community as whole move faster because you get more visibility, funding, man-hours dedicated to things that are on the leading edge of research. Consider a scenario where a researcher is required to explain exactly why some phenomenon happens. Maybe it took 1 year to find the original phenomenon and then it takes 10 years to explain it to a reasonable level. Everyone only gets the benefit of this research 11 years after. Now consider the opposite scenario. After 1 year the author publishes and gets the attention of fellow colleagues. Some of them will collaborate together adding more man-hours / year, reducing the total time. Some of them might have already discovered something similar and thus avoid all repeated work. Some of them might be better positioned to solve the explanation piece based on their field of expertise, personal interests, availability. All of this makes the innovation happen faster. What you often see (or should see in high quality papers) is an hypothesis of why something happens. This in itself is valuable. Many hypotheses are unproven until today. If you assume that these hypotheses are true you'll often find better results or find them faster; and if you don't you have discovered something interesting to report on.

loveparade 3y ago
I don't think we disagree. All I am saying is that if your goal is to filter papers by their likelihood of having a useful result, the best signal is the author and their reputation, not necessarily the content (unless there is perhaps some obviously amazing demo). We don't disagree that for the community as a whole the system of publishing early and often is much better than the alternative of imposing restrictions.
tpoacher 3y ago
This guys agree with you: https://www.octopus.ac/ https://www.octopus.ac/ Alas, whether their idea will withstand the current incentives structure in academia, remains to be seen.
psyklic 3y ago
I'm of the opposite view - publish only if you can prove it. If you reward half results, the field will be deluged with half truths. If someone publishes a hypothesis you're working on with only a half proof (and stands to gain the future credit), then there is little incentive to continue doing the work to prove it. This was actually a major issue one of my PhD advisors had, since it led to poor foundations for the field with little incentive to ensure their validity.
- rdedev 3y ago
  We need something in between. The paper author may not know why something is happening but has showed that it is statistically significant. Maybe he just does not have the context or background but someone else along the way. Of course the assumption is that people would do replication studies but no one is incentivized to do them so better to be on the safer side
low_tech_love 3y ago
That works well only if people have no motivation to fabricate or embellish results, and/or if they fully share the code and experimental setup. But in general I agree.

gumby 3y ago

> Also, in DL especially, it's quite difficult to perform ablation studies or get statistical significance numbers due experiments taking a long time to run. Aww, the poor babies. This the the nature of working with complex systems (life sciences, physics, social sciences, etc): teasing out signal from very noisy data that is quite expensive (in time at least) to obtain. On the engineering side things can (but don't always) move faster with a different kind of rigour. And that applies to observational reports in the sciences as well. All of which is to say that there's nothing wrong with the flood of papers that have a bunch of colored boxes and some running code. But consider those to be the equivalent of most of the reports in the 18th century Royal Society: "I have just returned from being a passenger on the HMS Foobar; while on the Malabar coast I saw an odd insect that looked like this drawing" And conversely spend referees' precious time on actual science.

godelski 3y ago
I mean I'm with you on the side of "ML needs more math" because we're about as sciencey as psychology, but I'm not quite on the "aww poor babies" side (yet). It's just the nature of how sciences evolve. The truth is that with deep rigorous theory, things like physics are easier. I'd say the same for psych too. It's a crazy amount of stats you need to do to account for all the unknowns and uncertainties but I'm not sure why these sciences push the theory people away when they can't even be bothered to add an epsilon to their models (which are always linear), but I digress. The thing that concerns me more though is the push against math. We see it a lot in ML but weirdly enough reviews will reject you if you don't have equations so you get tons of papers just copy pasting math that adds nothing to the paper. It's really weird and a clear indication of the laziness of reviews and area chairs. What I tell my students is "you don't need math to make good models but you need math to know why your models are wrong." This seems to be becoming more important than ever. Hell, how many people know about the Whitney embedding theorem? That itself helps answer half my undergrad questions and I think it's relatively unknown. But you need both. The thinkers and tinkerers. Referring to my longer rant, industry has captured the research field. I'm really happy they're involved but we're responding in insane ways. Let academia be the thinkers and let them make small models and don't expect a $100M budget for every paper. It's the railroading that annoys me and I think it's anti science. But no one wants to even acknowledge it's existence. I'll go ahead and say there's not much use to GPT papers unless you work for OpenAI (not to be confused with LLM papers). Let people explore new architectures and don't make them compete with sota models that have billions of dollars and man hours put into them. Hype is killing our field and it's hard to see because we're a high functioning addict.