Short reaction: No! In a little more detail, the author is talking about cases of "an amazing system" but, really, has in mind a quite narrow view of such sys

Short reaction: No! In a little more detail, the author is talking about cases of "an amazing system" but, really, has in mind a quite narrow view of such systems and how they work and can be, and have been, built. Really, the author is looking at less than, say, 5% of the great examples of how to build "an amazing system". In more detail: The article has its definition of AI: > In this article, I won't quibble over definitions, simply taking the broadest term as used in the media: artificial intelligence is whenever a system appears more intelligent than we expect it to be. Okay, from my usual understanding of AI, say, the peer-reviewed papers I published in AI, this is a relatively broad definition, but while considering this article, I'll accept that definition. But, oops, right away in the sentence before we have > If there's any promise I can make about the field of AI, So, the author is saying that AI is a field. From the author's definition of AI, that is a very broad field indeed! Maybe the most broad of all? Still I will try to consider the article: So, my next concern is > Ask what cases the system will fail on > "Revealing the failure cases of your models is a way of avoiding over-hype" > Kevin Murphy at ICML 2015 Now what is AI is starting to look quite narrow, narrow enough to have a "model" and "failure cases". So, oops: Consider a simple computer program that plays the little matchstick game Nim: There are two players. The game starts with three piles of matchsticks. For a move a player picks a pile and removes from it as many matchsticks as they want but at least one. The players alternate moves. The player who picks up the last matchstick loses. Okay, there is a simple algorithms for perfect play. Then if both players play perfect games, who wins depends just on who moves first. If only one player, A, is playing a perfect game but on that instance of the game with perfect play by the other player, B, should lose, the player A will still win as soon as player B makes a mistake. How to play a perfect game is in Courant and Robbins, What is Mathematics. So back to the last quote, there are no real "failure cases" and, really, no "model". Instead, there's some applied math. But, I'll keep reading: > No AI system yet developed will magically fit every use case. Well, the little algorithm for playing Nim fits "every use case". > If a researcher tells you that a model got state of the art results out of the box, they're either exceedingly lucky or giving you a fairy tale version of reality. No. As in a program to play perfect Nim, there is another explanation: They have a perfect solution for the problem. Again, by this far into the article, the author seems to have in mind a limited view of what AI is. > Even if their model is arbitrarily flexible, there are still fundamental limits placed on us by information theory. Not for a program written using the Courant and Robbins algorithm. Information theory has essentially nothing to do with it. Continuing: > "How much training data is required?" For the Nim program, none. > "Can this work unsupervised (= without labeling the examples)?" Well, since there are no examples, apparently so. > "Can the system predict out of vocabulary names?" WHAT? I doubt I understand the question. But with the Nim program, maybe so. The author seems to be assuming that the program is about natural language understanding. Not all computer programs are. > "How stable is the model's performance over time?" With the Nim program, perfectly stable. > Very few datasets are entirely representative of what the task looks like in the real world. "Dataset"? The Nim program doesn't have one. Again, it appears that the author is thinking of a narrow view of AI, much more narrow than the definition the author gave. > Any claim of advanced research without publication is suspect at best > "If you do research in isolation, the quality goes down. Which is why military research sucks." > Yann LeCun at ICML 2015 Uh, sorry, but I've done all my original research "in isolation" -- in a small, quiet room, sometimes on my back looking at the ceiling. For my Ph.D. dissertation in applied math, the first of the research was on an airplane ride from NY to TN. The rest of the research was independently in my first summer in graduate school. > The rate of change in the field of AI is such that anyone on the sidelines is at best keeping up. No, not really: I call my research applied mathematics, but it meets the author's definition of AI. But I pay essentially no serious attention to the field of AI (e.g., the work of people who claim to be in AI) at all. And the number of researchers in the field of AI, computer science, and information technology entrepreneurship that have the mathematical prerequisites to read my research is tiny. So, I'm not paying attention to the field of AI, and AI researchers are paying no attention to my work, some of which is published and, in one case, for one important problem, really, a significantly large set of important problems, totally knocks the socks off anything in AI. That case? Sure, some applied math, right, complete with theorems and proofs. In more detail, the work is some new results in mathematical statistics. > It is certainly not impossible for an entity to come out of nowhere with an amazing system but not it is far less likely. Naw! Essentially the opposite is true: If want "an amazing system", usually stay far away from AI -- just totally ignore the field. Instead, pursue applied math. E.g., molecular spectroscopy for identifying chemical molecules is terrific stuff, but is has essentially no "training data". Instead, the work is based on some quantum mechanics and group theory. Again, the author is thinking in a way quite narrow compared with the author's definition of AI and compared with what is done to build "an amazing system". > It also means they haven't been put through the standard evaluations that academia would have placed on them. Well, as I've seen in some of my peer-reviewed publications, my work in applied mathematics is more advanced and with standards commonly higher, e.g., theorems and proofs, than the academic review process is able to follow. Besides, any work I publish means I've given away the intellectual property, and now mostly I don't want to do that. > AI won't save a broken business plan. An easy upper bound is asking if the business plan would work with free human labor replacing the automated component. Going back decades, for many cases of "an amazing system", the idea that human labor could do the work at all is essentially absurd. Here the author is being narrow again. > Achieving human level performance on any real world task is an exceedingly difficult endeavor when moving away from a few simple and well defined tasks. No: Totally knocking the socks off human level performance is very common in applications of applied mathematics to complicated real problems, e.g., critical mass calculations for nuclear weapons, target detection via phased array passive sonar, also with Wiener filtering, to track the target, maybe with Kalman filtering, differential game theory for saying how a missile should catch a fighter plane, deterministic optimal control theory for, say, least time control to get an airplane to a given altitude, matching anti-ballistic missiles to incoming warheads, a huge range of applications of deterministic optimization, e.g., linear programming and integer linear programming, going way back to, say, even vacuum tube computers. > All of this is to say that if the business plan doesn't work with free humans, AI won't save it. Well, for my startup, there's no way humans could do the crucial, core applied mathematics, but my software based in my original applied math works just fine! Again the author is thinking about cases of "an amazing system" much more narrow than the author's definition of AI and much, much more narrow than good work in applied math going back at least 50 years.