3 ms·

> The result is, like the other poster says, a huge number of papers that claim to have solved really hard problems, which of course remain far from solved; in

> The result is, like the other poster says, a huge number of papers that claim to have solved really hard problems, which of course remain far from solved; in other words, so much useless noise. Could the answer be holding these papers to a stricter standard during peer review?

YeGoblynQueenne 8y ago
Machine learning researchers publish most of their work on Arxiv first (and often, only), so peer review will not stop wild claims from being publicised- and overhyped. The popular press helps with that as do blogs and youtube accounts that present the latest splashy paper for the lay audience (without, of course, any attempt at critical analysis). As to traditional publications in the field, these have often been criticised for their preference for work reporting high performance. In fact that's pretty much a requirement for publication in the most prestigious machine learning conferences and journals, to show improved performance against some previous work. This strongly motivates researchers to focus on one-off solutions to narrow problems, so that they can typeset one of those classic comparison tables with the best results highlighted, and claim a new record in some benchmark. This has now become the norm and it's difficult to see how it is going to change any time soon. Most probably the field will need to go through a serious crisis (an AI winter or something of that magnitude) before things seriously change.
YeGoblynQueenne 8y ago
Ah. To give a more controversial answer to your comment; you are asking, very reasonably: "isn't the solution to a deficit of scientific rigour, to increase scientific rigour"? Unfortunately, while machine learning is a very active research field that has contributed much technology, certainly to the industry but also occasionally to the sciences, it has been a long time since anyone has successfully accused it of science. There is not so much a deficit of scientific rigour, as a complete and utter disregard for it. Machine learning isn't science. It's a bunch of grown-up scientists banging their toy blocks together and gloating for having made the tallest tower. (there, I said it)

YeGoblynQueenne 8y ago

Machine learning researchers publish most of their work on Arxiv first (and often, only), so peer review will not stop wild claims from being publicised- and overhyped. The popular press helps with that as do blogs and youtube accounts that present the latest splashy paper for the lay audience (without, of course, any attempt at critical analysis). As to traditional publications in the field, these have often been criticised for their preference for work reporting high performance. In fact that's pretty much a requirement for publication in the most prestigious machine learning conferences and journals, to show improved performance against some previous work. This strongly motivates researchers to focus on one-off solutions to narrow problems, so that they can typeset one of those classic comparison tables with the best results highlighted, and claim a new record in some benchmark. This has now become the norm and it's difficult to see how it is going to change any time soon. Most probably the field will need to go through a serious crisis (an AI winter or something of that magnitude) before things seriously change.

YeGoblynQueenne 8y ago

Ah. To give a more controversial answer to your comment; you are asking, very reasonably: "isn't the solution to a deficit of scientific rigour, to increase scientific rigour"? Unfortunately, while machine learning is a very active research field that has contributed much technology, certainly to the industry but also occasionally to the sciences, it has been a long time since anyone has successfully accused it of science. There is not so much a deficit of scientific rigour, as a complete and utter disregard for it. Machine learning isn't science. It's a bunch of grown-up scientists banging their toy blocks together and gloating for having made the tallest tower. (there, I said it)