5 ms·

Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. DeepL today translates to and from Hungarian mu

Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. You can say it's pure scale, but whatever. Maybe that will be enough. And self supervision is working well nowadays through masked autoencoders and others, for pretraining. CLIP works great. Self driving exists and operates in certain locations. Yes, it's not everywhere yet. Image generation and chatbots are at a level unimaginable just a few years ago. The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. Also, someone has to do the work of designing things like Mask-RCNN or YOLO, their architecture is nontrivial. Anchor boxes, proposal networks, they had to be proposed. Batch normalization, Adam optimizer, learning rate schedules, initialization techniques like Glorot and He, ReLU replacing tanh, and a while host of other tricks. These were significant. Or take NeRF. If you are expecting such a discontinuous break as Newtonian mechanics, Darwinian evolution, relativity or quantum mechanics, you will be disappointed. Such moments are a handful examples over the entirety of human history. You can take it to an extreme when one scoffs at neural nets that it's nothing new. I mean Schmidhuber considers Gauss fitting a straight line by least squares in the 19th century to be an example of (single layer) neural net machine learning. Technically, sure. Maybe AI will be solved without any scientifically satisfying insight, but again, that has little consequence to the social transformation it will cause.

YeGoblynQueenne 3y ago
>> Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. You're the one who brought aesthetics into this, not me. I just want to know how things work, and why. That's what science means, to me. And the process we both, I think, understand is happening in machine learning research doesn't help anyone understand anything. One team publishes a paper claiming this one neat architectural trick beats such-and-such systems on such-and-such benchmarks. Three weeks later another team beats the former team on the same benchmark with another neat trick. All we learn is that some people came up with some neat tricks to beat benchmarks, and that you can't reuse the same trick twice. So what's the point of the trick, then, if you have to do it all from scratch, and you never know how to make something work? Neural nets work only when they work. When they don't work, nobody will tell you. Teams who beat the SOTA will gloat about it and keep schtum about all their many failures along the way. Their one neat trick can't beat _this_ benchmark? Nobody will ever know. The sum of knowledge generated by this process is exactly 0. That is why I say that progress is not being made. Not because it looks ugly. I don't care about elegant or ugly. I don't know why you assumed I do. I care about knowledge, that's my schtick. When there's a few dozen thousand people putting papers on arxiv that tell us nothing at all, that's a process that generates not knowledge, but noise. >> DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. Based on what? BLEU scores? >> The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. That's a rotten way to react to criticism: assume an emotional state in the critic and dismiss them as being dishonest. If you can easily dismiss the criticism by pointing out its flaws, do it, but if all you have to say is "nyah nyah you're butthurt" then you haven't dismissed anything.
- bonoboTP 3y ago
  It's hard to discuss without knowing where you're coming from and what kind of failures you've seen. There's is now enough practical best practice know how and rough process for how to structure an ML project and they tend to work. If you have some kind of vision task, like detecting and fitting a CAD model to some weird contraption nobody did before or some deep sea fish that was discovered yesterday, you can reliably get a neural net up and running to detect and segment and find it. Few-shot and zero-shot techniques exist. If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric. You see, when I and many other academicscomplain about academic culture, rewies and papers, what we complain about is that on the margins it's kind of nebulous and not at all clear cut whether the proposed method in those thousands of papers are each rigorously tested to a really convincing degree that they really beat the previous approach in some deeper sense, eg if the same amount of hyperparam search went to it, if a broader range of benchmarks were applied etc. But the presence of this noise does not crowd out the signal. Because there is a world outside the formalized systematic, officially presented Science with journals and conferences and awards and h indexes. There is a world of more immediate reality. The mechanized process of awards and publications and grants is not the primary object. I can tell if I'm hungry or full without looking at some figures of hormone tests from my blood. Yes, we have no guaranteed way to tell in advance what exact architecture and hyoerparameters will work best. Maybe the answer will be very context dependent. But this doesn't mean we are completely lost. The vocabulary of bias and variance, of overfitting, regularization etc do help. There are people who inspect loss landscapes for continuous reachability between different modes, there are hypotheses popping up regarding the sharpness of minima playing a role etc. Again I don't know how much you are inside the research community and how broadly you are aware of wide range of approaches that exists out there. The goal of "understanding why", and explainable AI are hot topics and there are good groups that come up with results, but none of it has a snappy one sentence summary where you slap your forehead and go "oh, so that's why it works". So yeah maybe we are in a pre-scientific phase of stumbling around, shaking the tree indiscriminately and picking up the fruits. We don't know where exactly the fruit is, but we keep being fed nevertheless. During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive. And it isn't guaranteed that the unreasonable effectiveness of math (a small number of simple and elegant formulas) also applies to this messy subject, as opposed to physics. Reality doesn't owe us any simple and satisfying explanations. I don't think it helps to complain that we don't get knowledge delivered. The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. The detractors like Gary Marcus never foresaw this state of things and was sure that things would plateau earlier. But that in itself was a hypothesis about the crowd of monkeys that researchers are, and whether their output will be such or such. And it turned out to falsify the pessimistic views of many. In any honest interpretation of what the early pioneers like Turing set out to do in terms of the Turing test for example, we have made real strides forward in capability. I know that you're eager for knowledge not just artifacts that work, but it's like complaining that a chess player won by making weird moves that are not listed in any textbooks and we have no idea why they worked. Whether we have a bearing on what exactly make our techniques work, they work.

YeGoblynQueenne 3y ago

>> Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. You're the one who brought aesthetics into this, not me. I just want to know how things work, and why. That's what science means, to me. And the process we both, I think, understand is happening in machine learning research doesn't help anyone understand anything. One team publishes a paper claiming this one neat architectural trick beats such-and-such systems on such-and-such benchmarks. Three weeks later another team beats the former team on the same benchmark with another neat trick. All we learn is that some people came up with some neat tricks to beat benchmarks, and that you can't reuse the same trick twice. So what's the point of the trick, then, if you have to do it all from scratch, and you never know how to make something work? Neural nets work only when they work. When they don't work, nobody will tell you. Teams who beat the SOTA will gloat about it and keep schtum about all their many failures along the way. Their one neat trick can't beat _this_ benchmark? Nobody will ever know. The sum of knowledge generated by this process is exactly 0. That is why I say that progress is not being made. Not because it looks ugly. I don't care about elegant or ugly. I don't know why you assumed I do. I care about knowledge, that's my schtick. When there's a few dozen thousand people putting papers on arxiv that tell us nothing at all, that's a process that generates not knowledge, but noise. >> DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. Based on what? BLEU scores? >> The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. That's a rotten way to react to criticism: assume an emotional state in the critic and dismiss them as being dishonest. If you can easily dismiss the criticism by pointing out its flaws, do it, but if all you have to say is "nyah nyah you're butthurt" then you haven't dismissed anything.

bonoboTP 3y ago
It's hard to discuss without knowing where you're coming from and what kind of failures you've seen. There's is now enough practical best practice know how and rough process for how to structure an ML project and they tend to work. If you have some kind of vision task, like detecting and fitting a CAD model to some weird contraption nobody did before or some deep sea fish that was discovered yesterday, you can reliably get a neural net up and running to detect and segment and find it. Few-shot and zero-shot techniques exist. If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric. You see, when I and many other academicscomplain about academic culture, rewies and papers, what we complain about is that on the margins it's kind of nebulous and not at all clear cut whether the proposed method in those thousands of papers are each rigorously tested to a really convincing degree that they really beat the previous approach in some deeper sense, eg if the same amount of hyperparam search went to it, if a broader range of benchmarks were applied etc. But the presence of this noise does not crowd out the signal. Because there is a world outside the formalized systematic, officially presented Science with journals and conferences and awards and h indexes. There is a world of more immediate reality. The mechanized process of awards and publications and grants is not the primary object. I can tell if I'm hungry or full without looking at some figures of hormone tests from my blood. Yes, we have no guaranteed way to tell in advance what exact architecture and hyoerparameters will work best. Maybe the answer will be very context dependent. But this doesn't mean we are completely lost. The vocabulary of bias and variance, of overfitting, regularization etc do help. There are people who inspect loss landscapes for continuous reachability between different modes, there are hypotheses popping up regarding the sharpness of minima playing a role etc. Again I don't know how much you are inside the research community and how broadly you are aware of wide range of approaches that exists out there. The goal of "understanding why", and explainable AI are hot topics and there are good groups that come up with results, but none of it has a snappy one sentence summary where you slap your forehead and go "oh, so that's why it works". So yeah maybe we are in a pre-scientific phase of stumbling around, shaking the tree indiscriminately and picking up the fruits. We don't know where exactly the fruit is, but we keep being fed nevertheless. During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive. And it isn't guaranteed that the unreasonable effectiveness of math (a small number of simple and elegant formulas) also applies to this messy subject, as opposed to physics. Reality doesn't owe us any simple and satisfying explanations. I don't think it helps to complain that we don't get knowledge delivered. The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. The detractors like Gary Marcus never foresaw this state of things and was sure that things would plateau earlier. But that in itself was a hypothesis about the crowd of monkeys that researchers are, and whether their output will be such or such. And it turned out to falsify the pessimistic views of many. In any honest interpretation of what the early pioneers like Turing set out to do in terms of the Turing test for example, we have made real strides forward in capability. I know that you're eager for knowledge not just artifacts that work, but it's like complaining that a chess player won by making weird moves that are not listed in any textbooks and we have no idea why they worked. Whether we have a bearing on what exactly make our techniques work, they work.