5 ms·

I've been in ML/vision since the HOG features + SVM and Viola-Jones face detection days, and have seen the methods run on non-cherry picked actual data. I remem

I've been in ML/vision since the HOG features + SVM and Viola-Jones face detection days, and have seen the methods run on non-cherry picked actual data. I remember when I first got reasonable image segmentation results on my own random photos, or when I first got good captions generated for my own holiday photos, not the test set of the same dataset. You see, back in the day datasets were extremely narrowly defined, small, with a particular style of lighting, the objects at a particular angle etc. The difference is so night-and-day that it doesn't need statistics or meticulous measurement to see it. A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has been the major catalyst of the field. But I'll offer up another one. There's some kind of shadow scholarship lurking in the background that doesn't have much of a footprint in the literature necessarily. This is of people trying each other's code, poking at stuff and gaining intuition and converging on practices that actually work, regardless of what the papers say. As an academic you develop a sixth sense for what numbers to ignore and what ideas to try perhaps. This is all vague and adhoc and not the clean story of what The Scientific Method is as described in the serious textbooks. But it does lead to the progress we see. It's just not elegant enough to put into papers. So the main story of the paper is often around a minor aspect of the work, because you can't easily make a story around "I sat down and really combed through everything and made sure not to have bugs and paid attention to all the small details others are careless about." But the net result is that the community will go forward with your codebase and your well-working setup. Even if they think that the part that makes it really good is that weird trick that the paper is selling, when that only contributed like 1%. And it's not like those long tail papers are totally bogus. Their methods surely work and would beat the state-of-the-art from ~2 years ago even under the highest standards of scrutiny. But it's doubtful it's really better than all the other ones that came out around the same time or a few months before. Overall, despite the shoddy practices required to publish, which introduces a lot of noise into the system, the trendline goes up, because there is genuine effort put into the work. Maybe all we are doing is some kind of unsexy distributed hyperparameter tuning and architecture search. Hinton humorously called it (at the single-person level) graduate student descent (i.e. that the phd student fiddles with the settings until it works well). And we may be doing this on a distributed level. But still, it does not seem to be overfitting, there have been studies on this that appear high quality. So well, maybe it's not a "science", but whatever else it is, it is working and does deliver results, so nobody feels a need to mess with a running system. Again, I don't insist on the label of science. Call it engineering, fiddling, or whatever, but the planes do land.

YeGoblynQueenne 3y ago
>> A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has been the major catalyst of the field. But the major catalyst to what? Neural nets today are just as capable as they were 40 years ago, only now we have bigger computers and more data and we can put more lipstick on the pig. I appreciate what you say, that systems today give better outputs than in earlier years. But you should also appreciate that those are essentially the same systems, with little tricks and hacks that have no clear contribution to performance, since as you say they are unreproducible. And each and every one of them inevitably, inexorably falls to bigger systems with more data and more compute - isn't that the bitter lesson after all? That you can try to be smart but the guy with the bigger machine and the bigger dataset will always win? And all the while the major limitation of neural nets trained with gradient optimisation remains unaddressed: that they can't function unless they have large amounts of data to train with and they only get performant enough to upload a paper to arxiv, let alone to use in a real problem, once they've been trained with hundreds of thousands, if not millions, of examples, at gigantic monetary cost. That is because -contrary to absurd claims by some- neural nets can't generalise their way out of a paper bag. If they could, they wouldn't need so many examples; they would learn, and generalise, robustly, from few examples. But they can't. Not at all. The field has simply made a virtue out of necessity, and hence comes the embrace of the magickal shibboleth of the Bitter Lesson. Neural nets can't generalise? Just overfit to more data. But, imagine if computer scientists, for example, claimed progress because we can now sort ever larger lists using bubblesort on giant compute clusters; or that the question of whether P = NP can now be laid to rest because we can solve simple instances of the Travelling Salesman Problem on a sufficiently powerful supercomputer. That is the Bitter Lesson applied to an imaginary computer science field, where progress had stalled, no sorting algorithms with good asymptotic complexity, like quicksort or mergesort, were known, and where combinatorial optimisation heuristics like Clause Driven Clause Learning where not invented. That is the machine learning field today, having learned the Bitter Lesson: a field that has given up on solving its research problems and instead turned to profit-making schemes to get more money from giant corporations who are in the business of beating people up with millions to make billions, and not in the business of answering any useful research questions. And you know what? Because of this, progress, even by the self-selected metric of "beats benchmarks", eventually stalls. Machine vision hasn't progressed much since 2012. Machine translation is still stuck in 2016 ish. Reinforcement Learning is coming next year, together with self-driving cars. Now we got LLMs and image generators. Give it a few years and that will be so 2020's and a new peak of hype will be reached, while LLMs remain incapable of telling truth from bullshit. Science is not an ideal, it's the way to make things work. Like, really work, in the long run. What you describe is the way to only make things sort of work and only in the near term.

YeGoblynQueenne 3y ago

>> A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has been the major catalyst of the field. But the major catalyst to what? Neural nets today are just as capable as they were 40 years ago, only now we have bigger computers and more data and we can put more lipstick on the pig. I appreciate what you say, that systems today give better outputs than in earlier years. But you should also appreciate that those are essentially the same systems, with little tricks and hacks that have no clear contribution to performance, since as you say they are unreproducible. And each and every one of them inevitably, inexorably falls to bigger systems with more data and more compute - isn't that the bitter lesson after all? That you can try to be smart but the guy with the bigger machine and the bigger dataset will always win? And all the while the major limitation of neural nets trained with gradient optimisation remains unaddressed: that they can't function unless they have large amounts of data to train with and they only get performant enough to upload a paper to arxiv, let alone to use in a real problem, once they've been trained with hundreds of thousands, if not millions, of examples, at gigantic monetary cost. That is because -contrary to absurd claims by some- neural nets can't generalise their way out of a paper bag. If they could, they wouldn't need so many examples; they would learn, and generalise, robustly, from few examples. But they can't. Not at all. The field has simply made a virtue out of necessity, and hence comes the embrace of the magickal shibboleth of the Bitter Lesson. Neural nets can't generalise? Just overfit to more data. But, imagine if computer scientists, for example, claimed progress because we can now sort ever larger lists using bubblesort on giant compute clusters; or that the question of whether P = NP can now be laid to rest because we can solve simple instances of the Travelling Salesman Problem on a sufficiently powerful supercomputer. That is the Bitter Lesson applied to an imaginary computer science field, where progress had stalled, no sorting algorithms with good asymptotic complexity, like quicksort or mergesort, were known, and where combinatorial optimisation heuristics like Clause Driven Clause Learning where not invented. That is the machine learning field today, having learned the Bitter Lesson: a field that has given up on solving its research problems and instead turned to profit-making schemes to get more money from giant corporations who are in the business of beating people up with millions to make billions, and not in the business of answering any useful research questions. And you know what? Because of this, progress, even by the self-selected metric of "beats benchmarks", eventually stalls. Machine vision hasn't progressed much since 2012. Machine translation is still stuck in 2016 ish. Reinforcement Learning is coming next year, together with self-driving cars. Now we got LLMs and image generators. Give it a few years and that will be so 2020's and a new peak of hype will be reached, while LLMs remain incapable of telling truth from bullshit. Science is not an ideal, it's the way to make things work. Like, really work, in the long run. What you describe is the way to only make things sort of work and only in the near term.