3 ms·

>> A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has be

>> A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has been the major catalyst of the field. But the major catalyst to what? Neural nets today are just as capable as they were 40 years ago, only now we have bigger computers and more data and we can put more lipstick on the pig. I appreciate what you say, that systems today give better outputs than in earlier years. But you should also appreciate that those are essentially the same systems, with little tricks and hacks that have no clear contribution to performance, since as you say they are unreproducible. And each and every one of them inevitably, inexorably falls to bigger systems with more data and more compute - isn't that the bitter lesson after all? That you can try to be smart but the guy with the bigger machine and the bigger dataset will always win? And all the while the major limitation of neural nets trained with gradient optimisation remains unaddressed: that they can't function unless they have large amounts of data to train with and they only get performant enough to upload a paper to arxiv, let alone to use in a real problem, once they've been trained with hundreds of thousands, if not millions, of examples, at gigantic monetary cost. That is because -contrary to absurd claims by some- neural nets can't generalise their way out of a paper bag. If they could, they wouldn't need so many examples; they would learn, and generalise, robustly, from few examples. But they can't. Not at all. The field has simply made a virtue out of necessity, and hence comes the embrace of the magickal shibboleth of the Bitter Lesson. Neural nets can't generalise? Just overfit to more data. But, imagine if computer scientists, for example, claimed progress because we can now sort ever larger lists using bubblesort on giant compute clusters; or that the question of whether P = NP can now be laid to rest because we can solve simple instances of the Travelling Salesman Problem on a sufficiently powerful supercomputer. That is the Bitter Lesson applied to an imaginary computer science field, where progress had stalled, no sorting algorithms with good asymptotic complexity, like quicksort or mergesort, were known, and where combinatorial optimisation heuristics like Clause Driven Clause Learning where not invented. That is the machine learning field today, having learned the Bitter Lesson: a field that has given up on solving its research problems and instead turned to profit-making schemes to get more money from giant corporations who are in the business of beating people up with millions to make billions, and not in the business of answering any useful research questions. And you know what? Because of this, progress, even by the self-selected metric of "beats benchmarks", eventually stalls. Machine vision hasn't progressed much since 2012. Machine translation is still stuck in 2016 ish. Reinforcement Learning is coming next year, together with self-driving cars. Now we got LLMs and image generators. Give it a few years and that will be so 2020's and a new peak of hype will be reached, while LLMs remain incapable of telling truth from bullshit. Science is not an ideal, it's the way to make things work. Like, really work, in the long run. What you describe is the way to only make things sort of work and only in the near term.

bonoboTP 3y ago
Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. You can say it's pure scale, but whatever. Maybe that will be enough. And self supervision is working well nowadays through masked autoencoders and others, for pretraining. CLIP works great. Self driving exists and operates in certain locations. Yes, it's not everywhere yet. Image generation and chatbots are at a level unimaginable just a few years ago. The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. Also, someone has to do the work of designing things like Mask-RCNN or YOLO, their architecture is nontrivial. Anchor boxes, proposal networks, they had to be proposed. Batch normalization, Adam optimizer, learning rate schedules, initialization techniques like Glorot and He, ReLU replacing tanh, and a while host of other tricks. These were significant. Or take NeRF. If you are expecting such a discontinuous break as Newtonian mechanics, Darwinian evolution, relativity or quantum mechanics, you will be disappointed. Such moments are a handful examples over the entirety of human history. You can take it to an extreme when one scoffs at neural nets that it's nothing new. I mean Schmidhuber considers Gauss fitting a straight line by least squares in the 19th century to be an example of (single layer) neural net machine learning. Technically, sure. Maybe AI will be solved without any scientifically satisfying insight, but again, that has little consequence to the social transformation it will cause.
- YeGoblynQueenne 3y ago
  >> Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. You're the one who brought aesthetics into this, not me. I just want to know how things work, and why. That's what science means, to me. And the process we both, I think, understand is happening in machine learning research doesn't help anyone understand anything. One team publishes a paper claiming this one neat architectural trick beats such-and-such systems on such-and-such benchmarks. Three weeks later another team beats the former team on the same benchmark with another neat trick. All we learn is that some people came up with some neat tricks to beat benchmarks, and that you can't reuse the same trick twice. So what's the point of the trick, then, if you have to do it all from scratch, and you never know how to make something work? Neural nets work only when they work. When they don't work, nobody will tell you. Teams who beat the SOTA will gloat about it and keep schtum about all their many failures along the way. Their one neat trick can't beat _this_ benchmark? Nobody will ever know. The sum of knowledge generated by this process is exactly 0. That is why I say that progress is not being made. Not because it looks ugly. I don't care about elegant or ugly. I don't know why you assumed I do. I care about knowledge, that's my schtick. When there's a few dozen thousand people putting papers on arxiv that tell us nothing at all, that's a process that generates not knowledge, but noise. >> DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. Based on what? BLEU scores? >> The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. That's a rotten way to react to criticism: assume an emotional state in the critic and dismiss them as being dishonest. If you can easily dismiss the criticism by pointing out its flaws, do it, but if all you have to say is "nyah nyah you're butthurt" then you haven't dismissed anything.

bonoboTP 3y ago

Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. You can say it's pure scale, but whatever. Maybe that will be enough. And self supervision is working well nowadays through masked autoencoders and others, for pretraining. CLIP works great. Self driving exists and operates in certain locations. Yes, it's not everywhere yet. Image generation and chatbots are at a level unimaginable just a few years ago. The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. Also, someone has to do the work of designing things like Mask-RCNN or YOLO, their architecture is nontrivial. Anchor boxes, proposal networks, they had to be proposed. Batch normalization, Adam optimizer, learning rate schedules, initialization techniques like Glorot and He, ReLU replacing tanh, and a while host of other tricks. These were significant. Or take NeRF. If you are expecting such a discontinuous break as Newtonian mechanics, Darwinian evolution, relativity or quantum mechanics, you will be disappointed. Such moments are a handful examples over the entirety of human history. You can take it to an extreme when one scoffs at neural nets that it's nothing new. I mean Schmidhuber considers Gauss fitting a straight line by least squares in the 19th century to be an example of (single layer) neural net machine learning. Technically, sure. Maybe AI will be solved without any scientifically satisfying insight, but again, that has little consequence to the social transformation it will cause.

YeGoblynQueenne 3y ago
>> Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. You're the one who brought aesthetics into this, not me. I just want to know how things work, and why. That's what science means, to me. And the process we both, I think, understand is happening in machine learning research doesn't help anyone understand anything. One team publishes a paper claiming this one neat architectural trick beats such-and-such systems on such-and-such benchmarks. Three weeks later another team beats the former team on the same benchmark with another neat trick. All we learn is that some people came up with some neat tricks to beat benchmarks, and that you can't reuse the same trick twice. So what's the point of the trick, then, if you have to do it all from scratch, and you never know how to make something work? Neural nets work only when they work. When they don't work, nobody will tell you. Teams who beat the SOTA will gloat about it and keep schtum about all their many failures along the way. Their one neat trick can't beat _this_ benchmark? Nobody will ever know. The sum of knowledge generated by this process is exactly 0. That is why I say that progress is not being made. Not because it looks ugly. I don't care about elegant or ugly. I don't know why you assumed I do. I care about knowledge, that's my schtick. When there's a few dozen thousand people putting papers on arxiv that tell us nothing at all, that's a process that generates not knowledge, but noise. >> DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. Based on what? BLEU scores? >> The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. That's a rotten way to react to criticism: assume an emotional state in the critic and dismiss them as being dishonest. If you can easily dismiss the criticism by pointing out its flaws, do it, but if all you have to say is "nyah nyah you're butthurt" then you haven't dismissed anything.