5 ms·

Are We Making Much Progress? Analysis of Recent Neural Recommendation Approaches

86 points by sndean 7y ago

azhenley 7y ago
To answer if we are making much progress, here’s a quote: “Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method.“
- kariluoma 7y ago
  The method that outperformed the others was MultVAE [1] [1]: https://arxiv.org/abs/1802.05814 https://arxiv.org/abs/1802.05814
yorwba 7y ago
Previously: https://news.ycombinator.com/item?id=20495047 https://news.ycombinator.com/item?id=20495047
olooney 7y ago
Eerily reminiscent of the replication crisis in psychology and the social sciences. My key takeaways: 1) Half the papers couldn't be reproduced on a technical level. Publish your code and your data, people! 2) Most of these papers uses "weak baselines" so they can show some kind of improvement and get their paper published. I'm conflicted about this because if we require every paper to beat state-of-the-art, we'd (collectively, as the entire discipline) be lucky to publish one paper a year. From one point of view, these papers actually represent a form of publishing negative results - we tried this and it didn't work - which isn't a bad thing. But the biased way its presented makes it harder to separate the wheat from the chaff. 3) It's not obvious that we're going to squeeze any more business value from this particular stone. Sometimes all the useful information in a dataset can be found with a fairly simple algorithm. Not everything benefits from a more complex representation, and sometimes you can't fix that with regularization or more data. Sometimes you just have to use the simple model and accept that it captures all the signal that's available and everything else is noise.
- SubiculumCode 7y ago
  Not all papers are about achieving state of the art performance,but to explore the general viability of a new approach.
- mattkrause 7y ago
  It's not just reminiscent. It's the same incentive structure: people need papers, as many and as soon as possible. The results need to sound good (so they land in good venues), but they won't be scrutinized too much once they do.
- currymj 7y ago
  regarding 1, even papers with high quality published code fall prey to bit rot very quickly -- go dig through some git repos from ML papers from 2015 and try to run them. distributing Docker containers works better but even that won't necessarily help, because especially if you have to use GPUs the container software has been shifting underneath you. plus it's still cumbersome to make a fair comparison between containerized code and a new approach implemented somewhere else, especially if it's on different data. Would it be worth making everyone expend a huge amount of unpaid additional effort maintaining the code for every ML paper ever published, for years to come? I'm not sure -- many are not very good and should probably be left to die. On the other hand it's common and immensely frustrating to be unable to reproduce results. regarding 2, it seems like if the technique itself works in a substantially different way from the state of the art, even if it can't consistently beat it, that's still a positive result -- something new.
  - amelius 7y ago
    What if, instead of publishing just the code, authors would publish intermediate results (e.g. tensors) of running the code? So basically sprinkle the code with print statements. Then if a future researcher has difficulty reproducing the result, they know exactly at which step the problem occurs.
    - newen 7y ago
      Haha that's not going to work for typical models that have millions of parameters. Besides, it's not the code that's important (except for verifying that it's a good method) it's the actual method, it's description, why it works, how it works, how it can be improved, adpated, etc.
anjc 7y ago
There are fields (recommender systems in particular in my opinion) where you will not get published if you report a strong baseline and your approach does not outperform it, but you may get published if you use weak baselines and your approach outperforms it. This doesn't make sense to me as surely it's possible for promising new approaches to not yet outperform current state of the art methods. These peer review aberrations incentivise these bad research practices I think.
joker3 7y ago
Most papers in most disciplines end up not being worth much, and the incentives regarding publishing hold across all of computer science. Is there really something particularly bad going on in deep learning? Or is it just the usual process?
- newen 7y ago
  Most papers are slight improvements, or barely an improvement after selectively comparing against other methods or using particular datasets that work well with that method, or after mangling the results so much with vagaries and statistics that the results look kind of good if you squint a lot. That's just how it is because 99% of academics are not geniuses but they still need to graduate or get their tenure. But the papers of non-geniuses provide ideas and an environment for the geniuses to publish their work and flourish, so it works out in the end.

86 points by sndean 7y ago

To answer if we are making much progress, here’s a quote: “Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method.“

kariluoma 7y ago
The method that outperformed the others was MultVAE [1] [1]: https://arxiv.org/abs/1802.05814 https://arxiv.org/abs/1802.05814

yorwba 7y ago

Previously: https://news.ycombinator.com/item?id=20495047 https://news.ycombinator.com/item?id=20495047

olooney 7y ago

Eerily reminiscent of the replication crisis in psychology and the social sciences. My key takeaways: 1) Half the papers couldn't be reproduced on a technical level. Publish your code and your data, people! 2) Most of these papers uses "weak baselines" so they can show some kind of improvement and get their paper published. I'm conflicted about this because if we require every paper to beat state-of-the-art, we'd (collectively, as the entire discipline) be lucky to publish one paper a year. From one point of view, these papers actually represent a form of publishing negative results - we tried this and it didn't work - which isn't a bad thing. But the biased way its presented makes it harder to separate the wheat from the chaff. 3) It's not obvious that we're going to squeeze any more business value from this particular stone. Sometimes all the useful information in a dataset can be found with a fairly simple algorithm. Not everything benefits from a more complex representation, and sometimes you can't fix that with regularization or more data. Sometimes you just have to use the simple model and accept that it captures all the signal that's available and everything else is noise.

SubiculumCode 7y ago
Not all papers are about achieving state of the art performance,but to explore the general viability of a new approach.
mattkrause 7y ago
It's not just reminiscent. It's the same incentive structure: people need papers, as many and as soon as possible. The results need to sound good (so they land in good venues), but they won't be scrutinized too much once they do.
currymj 7y ago
regarding 1, even papers with high quality published code fall prey to bit rot very quickly -- go dig through some git repos from ML papers from 2015 and try to run them. distributing Docker containers works better but even that won't necessarily help, because especially if you have to use GPUs the container software has been shifting underneath you. plus it's still cumbersome to make a fair comparison between containerized code and a new approach implemented somewhere else, especially if it's on different data. Would it be worth making everyone expend a huge amount of unpaid additional effort maintaining the code for every ML paper ever published, for years to come? I'm not sure -- many are not very good and should probably be left to die. On the other hand it's common and immensely frustrating to be unable to reproduce results. regarding 2, it seems like if the technique itself works in a substantially different way from the state of the art, even if it can't consistently beat it, that's still a positive result -- something new.
- amelius 7y ago
  What if, instead of publishing just the code, authors would publish intermediate results (e.g. tensors) of running the code? So basically sprinkle the code with print statements. Then if a future researcher has difficulty reproducing the result, they know exactly at which step the problem occurs.
  - newen 7y ago
    Haha that's not going to work for typical models that have millions of parameters. Besides, it's not the code that's important (except for verifying that it's a good method) it's the actual method, it's description, why it works, how it works, how it can be improved, adpated, etc.

anjc 7y ago

There are fields (recommender systems in particular in my opinion) where you will not get published if you report a strong baseline and your approach does not outperform it, but you may get published if you use weak baselines and your approach outperforms it. This doesn't make sense to me as surely it's possible for promising new approaches to not yet outperform current state of the art methods. These peer review aberrations incentivise these bad research practices I think.

joker3 7y ago

Most papers in most disciplines end up not being worth much, and the incentives regarding publishing hold across all of computer science. Is there really something particularly bad going on in deep learning? Or is it just the usual process?

newen 7y ago
Most papers are slight improvements, or barely an improvement after selectively comparing against other methods or using particular datasets that work well with that method, or after mangling the results so much with vagaries and statistics that the results look kind of good if you squint a lot. That's just how it is because 99% of academics are not geniuses but they still need to graduate or get their tenure. But the papers of non-geniuses provide ideas and an environment for the geniuses to publish their work and flourish, so it works out in the end.