3 ms·

>> Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. You're the one who brought aesthetics into t

>> Again I share the aesthetics distaste to how this progress looks. But one has to get over such personal tastes. You're the one who brought aesthetics into this, not me. I just want to know how things work, and why. That's what science means, to me. And the process we both, I think, understand is happening in machine learning research doesn't help anyone understand anything. One team publishes a paper claiming this one neat architectural trick beats such-and-such systems on such-and-such benchmarks. Three weeks later another team beats the former team on the same benchmark with another neat trick. All we learn is that some people came up with some neat tricks to beat benchmarks, and that you can't reuse the same trick twice. So what's the point of the trick, then, if you have to do it all from scratch, and you never know how to make something work? Neural nets work only when they work. When they don't work, nobody will tell you. Teams who beat the SOTA will gloat about it and keep schtum about all their many failures along the way. Their one neat trick can't beat _this_ benchmark? Nobody will ever know. The sum of knowledge generated by this process is exactly 0. That is why I say that progress is not being made. Not because it looks ugly. I don't care about elegant or ugly. I don't know why you assumed I do. I care about knowledge, that's my schtick. When there's a few dozen thousand people putting papers on arxiv that tell us nothing at all, that's a process that generates not knowledge, but noise. >> DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. Based on what? BLEU scores? >> The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane. That's a rotten way to react to criticism: assume an emotional state in the critic and dismiss them as being dishonest. If you can easily dismiss the criticism by pointing out its flaws, do it, but if all you have to say is "nyah nyah you're butthurt" then you haven't dismissed anything.

bonoboTP 3y ago
It's hard to discuss without knowing where you're coming from and what kind of failures you've seen. There's is now enough practical best practice know how and rough process for how to structure an ML project and they tend to work. If you have some kind of vision task, like detecting and fitting a CAD model to some weird contraption nobody did before or some deep sea fish that was discovered yesterday, you can reliably get a neural net up and running to detect and segment and find it. Few-shot and zero-shot techniques exist. If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric. You see, when I and many other academicscomplain about academic culture, rewies and papers, what we complain about is that on the margins it's kind of nebulous and not at all clear cut whether the proposed method in those thousands of papers are each rigorously tested to a really convincing degree that they really beat the previous approach in some deeper sense, eg if the same amount of hyperparam search went to it, if a broader range of benchmarks were applied etc. But the presence of this noise does not crowd out the signal. Because there is a world outside the formalized systematic, officially presented Science with journals and conferences and awards and h indexes. There is a world of more immediate reality. The mechanized process of awards and publications and grants is not the primary object. I can tell if I'm hungry or full without looking at some figures of hormone tests from my blood. Yes, we have no guaranteed way to tell in advance what exact architecture and hyoerparameters will work best. Maybe the answer will be very context dependent. But this doesn't mean we are completely lost. The vocabulary of bias and variance, of overfitting, regularization etc do help. There are people who inspect loss landscapes for continuous reachability between different modes, there are hypotheses popping up regarding the sharpness of minima playing a role etc. Again I don't know how much you are inside the research community and how broadly you are aware of wide range of approaches that exists out there. The goal of "understanding why", and explainable AI are hot topics and there are good groups that come up with results, but none of it has a snappy one sentence summary where you slap your forehead and go "oh, so that's why it works". So yeah maybe we are in a pre-scientific phase of stumbling around, shaking the tree indiscriminately and picking up the fruits. We don't know where exactly the fruit is, but we keep being fed nevertheless. During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive. And it isn't guaranteed that the unreasonable effectiveness of math (a small number of simple and elegant formulas) also applies to this messy subject, as opposed to physics. Reality doesn't owe us any simple and satisfying explanations. I don't think it helps to complain that we don't get knowledge delivered. The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. The detractors like Gary Marcus never foresaw this state of things and was sure that things would plateau earlier. But that in itself was a hypothesis about the crowd of monkeys that researchers are, and whether their output will be such or such. And it turned out to falsify the pessimistic views of many. In any honest interpretation of what the early pioneers like Turing set out to do in terms of the Turing test for example, we have made real strides forward in capability. I know that you're eager for knowledge not just artifacts that work, but it's like complaining that a chess player won by making weird moves that are not listed in any textbooks and we have no idea why they worked. Whether we have a bearing on what exactly make our techniques work, they work.
- YeGoblynQueenne 3y ago
  Regarding my background I just finished my PhD and I'm doing a postdoc on explainable machine learning for robotics. My PhD was on Inductive Logic Programming which is a virtually unknown form of machine learning of logic theories, but I have a background in statistical AI, neural nets and NLP, from my Master's days. Btw, lest this causes confusion, my part in the "explainable machine learning for robotics" project is to machine-learn autonomous behaviours for an underwater search-and-rescue robot. My specialty is not explainability, it's rather learning ability I'm interested in. That out the way, >> If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric. No, sorry. I want a metric I can trust and not one that is equivalent to good, old-fashioned eyballing. If I wanted to eyball, I got eyballs, I can eyball. I want a metric that doesn't depend on someone's subjective evaluation. Unfortunately, the day that someone will come up with an objective test for machine translation is the day we'd have solved machine translation, and possibly all of AI, to boot. Btw, I studied NLP at the time when Google Translate had just switched to Neural Machine Translation. I'm Greek, and I remember on the one hand the dithyrambic self-congratulations on Google blogs, and on the other hand how ridiculously crap Google Translate was in translating from and to Greek. And it still is, almost ten years later (My MSc was in 2014). I also have a cluster of friends and acquaintances who are all professional translators and interpreters and I know that they are still ripped off by translation agencies who give them automatic translations claiming that they only need an hour of work to correct, when they need maybe a day or two. That kind of test will also not do. "Progress" in machine learning means that every few years the hype peaks for yet another AI fountain of youth, or AI El Dorado, then that loses steam, and there's some sanity for a few years until the net thing comes along and trends on social media, then everybody starts writing super-serious papers about it sounding all business-like and scientifically minded, until Google, or OpenAI now, or some other big gun releases an implementation and everyone has to run behind them like idiots. It's basically science by twitter and it's completely bogus. Bottom line, you want to make machine lerning work? Be Google. >> The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. To what? I think I've made myself very clear that I don't think we've got anywhere worth talking about. So my computer can now talk back to me. So what? Why is this a scientific achievement? Why does this answer scientific questions? Which ones? >> During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive. I've heard this before but I don't know any examples where technological progress did not go hand-in-hand with scientific understanding. For example, when the Wright brothers made a machine that could fly, the followed on the footsteps of probably hundreds of others who had done the work of science, developing such necessary concepts as aerodynamics- like George Cayley, Otto Lilienthal of course, and in fact the Wrights themselves had set up a wind tunnel and studied the aerodynamics of their machines. The difference with AI of course is that it's much easier to point to a thing and say "it flies". But that's another matter. Lest I be misunderstood: I'm enjoying our exchange. Thanks.

bonoboTP 3y ago

It's hard to discuss without knowing where you're coming from and what kind of failures you've seen. There's is now enough practical best practice know how and rough process for how to structure an ML project and they tend to work. If you have some kind of vision task, like detecting and fitting a CAD model to some weird contraption nobody did before or some deep sea fish that was discovered yesterday, you can reliably get a neural net up and running to detect and segment and find it. Few-shot and zero-shot techniques exist. If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric. You see, when I and many other academicscomplain about academic culture, rewies and papers, what we complain about is that on the margins it's kind of nebulous and not at all clear cut whether the proposed method in those thousands of papers are each rigorously tested to a really convincing degree that they really beat the previous approach in some deeper sense, eg if the same amount of hyperparam search went to it, if a broader range of benchmarks were applied etc. But the presence of this noise does not crowd out the signal. Because there is a world outside the formalized systematic, officially presented Science with journals and conferences and awards and h indexes. There is a world of more immediate reality. The mechanized process of awards and publications and grants is not the primary object. I can tell if I'm hungry or full without looking at some figures of hormone tests from my blood. Yes, we have no guaranteed way to tell in advance what exact architecture and hyoerparameters will work best. Maybe the answer will be very context dependent. But this doesn't mean we are completely lost. The vocabulary of bias and variance, of overfitting, regularization etc do help. There are people who inspect loss landscapes for continuous reachability between different modes, there are hypotheses popping up regarding the sharpness of minima playing a role etc. Again I don't know how much you are inside the research community and how broadly you are aware of wide range of approaches that exists out there. The goal of "understanding why", and explainable AI are hot topics and there are good groups that come up with results, but none of it has a snappy one sentence summary where you slap your forehead and go "oh, so that's why it works". So yeah maybe we are in a pre-scientific phase of stumbling around, shaking the tree indiscriminately and picking up the fruits. We don't know where exactly the fruit is, but we keep being fed nevertheless. During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive. And it isn't guaranteed that the unreasonable effectiveness of math (a small number of simple and elegant formulas) also applies to this messy subject, as opposed to physics. Reality doesn't owe us any simple and satisfying explanations. I don't think it helps to complain that we don't get knowledge delivered. The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. The detractors like Gary Marcus never foresaw this state of things and was sure that things would plateau earlier. But that in itself was a hypothesis about the crowd of monkeys that researchers are, and whether their output will be such or such. And it turned out to falsify the pessimistic views of many. In any honest interpretation of what the early pioneers like Turing set out to do in terms of the Turing test for example, we have made real strides forward in capability. I know that you're eager for knowledge not just artifacts that work, but it's like complaining that a chess player won by making weird moves that are not listed in any textbooks and we have no idea why they worked. Whether we have a bearing on what exactly make our techniques work, they work.

YeGoblynQueenne 3y ago
Regarding my background I just finished my PhD and I'm doing a postdoc on explainable machine learning for robotics. My PhD was on Inductive Logic Programming which is a virtually unknown form of machine learning of logic theories, but I have a background in statistical AI, neural nets and NLP, from my Master's days. Btw, lest this causes confusion, my part in the "explainable machine learning for robotics" project is to machine-learn autonomous behaviours for an underwater search-and-rescue robot. My specialty is not explainability, it's rather learning ability I'm interested in. That out the way, >> If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric. No, sorry. I want a metric I can trust and not one that is equivalent to good, old-fashioned eyballing. If I wanted to eyball, I got eyballs, I can eyball. I want a metric that doesn't depend on someone's subjective evaluation. Unfortunately, the day that someone will come up with an objective test for machine translation is the day we'd have solved machine translation, and possibly all of AI, to boot. Btw, I studied NLP at the time when Google Translate had just switched to Neural Machine Translation. I'm Greek, and I remember on the one hand the dithyrambic self-congratulations on Google blogs, and on the other hand how ridiculously crap Google Translate was in translating from and to Greek. And it still is, almost ten years later (My MSc was in 2014). I also have a cluster of friends and acquaintances who are all professional translators and interpreters and I know that they are still ripped off by translation agencies who give them automatic translations claiming that they only need an hour of work to correct, when they need maybe a day or two. That kind of test will also not do. "Progress" in machine learning means that every few years the hype peaks for yet another AI fountain of youth, or AI El Dorado, then that loses steam, and there's some sanity for a few years until the net thing comes along and trends on social media, then everybody starts writing super-serious papers about it sounding all business-like and scientifically minded, until Google, or OpenAI now, or some other big gun releases an implementation and everyone has to run behind them like idiots. It's basically science by twitter and it's completely bogus. Bottom line, you want to make machine lerning work? Be Google. >> The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. To what? I think I've made myself very clear that I don't think we've got anywhere worth talking about. So my computer can now talk back to me. So what? Why is this a scientific achievement? Why does this answer scientific questions? Which ones? >> During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive. I've heard this before but I don't know any examples where technological progress did not go hand-in-hand with scientific understanding. For example, when the Wright brothers made a machine that could fly, the followed on the footsteps of probably hundreds of others who had done the work of science, developing such necessary concepts as aerodynamics- like George Cayley, Otto Lilienthal of course, and in fact the Wrights themselves had set up a wind tunnel and studied the aerodynamics of their machines. The difference with AI of course is that it's much easier to point to a thing and say "it flies". But that's another matter. Lest I be misunderstood: I'm enjoying our exchange. Thanks.