3 ms·

I have been in the industry for sometime, and could never fathom the pessimism of many who put a major AI revolution out toward the end of the century or beyond

I have been in the industry for sometime, and could never fathom the pessimism of many who put a major AI revolution out toward the end of the century or beyond. But 2022 was a remarkable year of progress, even for optimists. In part, what is so remarkable is how many qualitative improvements occurred in chat (and other modalities) with fairly mundane improvements in algorithms. And vast computing resources of course. Add in multiple modalities of text, voice, vision, kinematics, decision making, ... and some algorithm improvements to replace the "reset" per chat/interaction session with something that can accumulate context and experience in an open open ended sequence ... The flywheel is accelerating. These dramatic successes have thrown open the floodgates of resources and researchers. The heartbeat for breakthroughs is now between months or a year or so, instead of decades. Things are looking crazy.

YeGoblynQueenne 4y ago
All that sounds very dramatic, but I am unsure what people mean when they say they have "been in the industry", like you say. I don't wish to interrogate you (or "cross-examine" you as the guidelines say), so I'll list my background instead. I've graduated in 2011 in CS, did a Master's in data science in 2014, and a PhD in AI in 2017 (I'm just finishing the revisions to my thesis). In my Master's I focused on NLP and neural nets, and I've done a fair bit of language generation over the years, both statistical and not (i.e. grammars). My opinion, informed by my background, is that what we are seeing should not be a surprise. That doesn't mean that it is all to be expected. We've had language models for a very long time, and they have certainly gotten better and bigger (although mainly bigger). But it has never been easy to predict what the behaviour of a language model should be. It has always been difficult to form a clear expectation of what a language model should be capable of- let alone one as big, and as smooth (in the statistical sense, of smooth vectors) as the large language models that started appearing a few years ago (ca. 2016 ish). So I don't think that surprise is justified. There is a general lack of experience that is causing people to think they have formed expectations, when they haven't, who then suddendly find themselves in the middle of an unknown situation and don't know how to react. But it's still my impression that most people who have good knowledge of NLP and language generation are not, currently, losing their heads with GPT-stuff.
- Nevermark 4y ago
  I am not sure how I would or should describe my history in the field here so I won’t. I am still very active, but now on a different axis of progress from gradient based modeling. So your “recent” (to me) perspective is interesting to me. For me there has been multiple clear upticks in the rate of progress in the last few years. Some of it insights, some of it simply resources and efforts. But together the results are clearly moving faster and faster, as in an acceleration of acceleration. It may be that it’s just an especially productive moment, in AI. But AI, like transistors, is a self reinforcing technology, so I think the big AI explosion is really happening. I would be very surprised if we don’t have general intelligences exceeding humans within a few years now. Certainly by 2030. My “surprise” isn’t so much the specific improvements in chat the last year. But how fast advances in that order are now happening relative to the long slow accumulation of progress in preceding decades and how much of the new process is enabled just by provisioning more resources. It’s kind of ridiculous in the sense that many deep learning tools are not even using all the best gradient optimization algorithms and techniques, but progress in results is happening so fast, that algorithmic waste isn’t even an issue. There is so much slack algorithmically, that big progress is already a forgone conclusion even if the tool builders were just carefully folding in past knowledge, or just iterating on straightforward efficiency improvements, and not making headway on new innovations. I haven’t ever seen a field hit this state. And maybe having a few decades of experience with a field makes a radical change in the rate of progress, and the specifics of that, easier to see. Do you have the sense that general AI is now within a decade or so or less? The variety of glass half full, vs. glass half empty judgements around every advance is interesting to me. I find different people have very different sensitivities to constant steps vs. the first and second derivatives of progress they are signaling. 2022 saw more algorithmic progress in poetry, essay writing, graphic art, debate, joke construction, etc, etc, etc. than a human could make in … well a lot more than a year. That’s a new red line we have now crossed. That’s faster than human process. Much faster.
  - YeGoblynQueenne 4y ago
    Thanks for the clarifications. >> Do you have the sense that general AI is now within a decade or so or less? Not at all. My impression is that nobody is really working on creating general AI, and that if you take the field of machine learning research as a whole, if the purpose ever was to create general AI (which it never was), the field would be currently running backwards, far away from any kind of "intelligence", or any kind of generality at all. It is beyond clear to me that any sort of generalisation cannot be achieved by neural nets trained with gradient optimisation. That's because gradient optimisation will always, always get stuck at local optima. As you seem to know yourself, gradient optimisation has been around for many decades and I reckon there's been quite a few attempts to see if it can avoid getting stuck at local optima. And yet, well, it still gets stuck at local optima. That's probably its strength anyway, although it's also how it is self-limited. So there's only so much you can do when training with gradient optimisation. There is nothing to do about it. There is nobody who can do anything about it. All the billions of research, all the hordes of researchers, cannot get Humpty Dumpty unstuck from local optima. And so there's not a chance of a snowball in hell that general AI is going to be anywhere near where we get to in a hundred years from now, with current approaches. >> I haven’t ever seen a field hit this state. Well, I agree, but I disagree fundamentally. For me, the state of the field is deplorable! "The field" of machine learning research, that is. That field has abandoned every semblance of doing science and it has become a complete circus, where all that matters is what is the next big company that's paying billions for researchers who know how to train a neural net. Just look at the papers. Thousands of papers are placed on arxiv every month. Hundreds are even published. The vast, vast majority of them do nothing useful at all, and just claim to have moved the state of the art forward a couple of percentage points. They beat a benchmark, brag about it, and go home happy that they will be remembered... for a couple days at least, until someone else with no ideas breaks their record. It's a glorified Kaggle competition, and a free for all, but it achieves nothing. There are no theoretical results in any of the papers, except those of the very few people left at the top of the mess, who would actually know science if it bit them in the leg. There is nothing but noise. And if anyone comes up with a novel idea, in all this mess, that novel idea will be buried under a ton of dreary, miserable dross. Or, to cut this short, what you are seeing is supply and demand: big tech corps supply millions, graduate students run towards them with open arms to get the millions. Progress? What, in that situation? Not really.

YeGoblynQueenne 4y ago

All that sounds very dramatic, but I am unsure what people mean when they say they have "been in the industry", like you say. I don't wish to interrogate you (or "cross-examine" you as the guidelines say), so I'll list my background instead. I've graduated in 2011 in CS, did a Master's in data science in 2014, and a PhD in AI in 2017 (I'm just finishing the revisions to my thesis). In my Master's I focused on NLP and neural nets, and I've done a fair bit of language generation over the years, both statistical and not (i.e. grammars). My opinion, informed by my background, is that what we are seeing should not be a surprise. That doesn't mean that it is all to be expected. We've had language models for a very long time, and they have certainly gotten better and bigger (although mainly bigger). But it has never been easy to predict what the behaviour of a language model should be. It has always been difficult to form a clear expectation of what a language model should be capable of- let alone one as big, and as smooth (in the statistical sense, of smooth vectors) as the large language models that started appearing a few years ago (ca. 2016 ish). So I don't think that surprise is justified. There is a general lack of experience that is causing people to think they have formed expectations, when they haven't, who then suddendly find themselves in the middle of an unknown situation and don't know how to react. But it's still my impression that most people who have good knowledge of NLP and language generation are not, currently, losing their heads with GPT-stuff.

Nevermark 4y ago
I am not sure how I would or should describe my history in the field here so I won’t. I am still very active, but now on a different axis of progress from gradient based modeling. So your “recent” (to me) perspective is interesting to me. For me there has been multiple clear upticks in the rate of progress in the last few years. Some of it insights, some of it simply resources and efforts. But together the results are clearly moving faster and faster, as in an acceleration of acceleration. It may be that it’s just an especially productive moment, in AI. But AI, like transistors, is a self reinforcing technology, so I think the big AI explosion is really happening. I would be very surprised if we don’t have general intelligences exceeding humans within a few years now. Certainly by 2030. My “surprise” isn’t so much the specific improvements in chat the last year. But how fast advances in that order are now happening relative to the long slow accumulation of progress in preceding decades and how much of the new process is enabled just by provisioning more resources. It’s kind of ridiculous in the sense that many deep learning tools are not even using all the best gradient optimization algorithms and techniques, but progress in results is happening so fast, that algorithmic waste isn’t even an issue. There is so much slack algorithmically, that big progress is already a forgone conclusion even if the tool builders were just carefully folding in past knowledge, or just iterating on straightforward efficiency improvements, and not making headway on new innovations. I haven’t ever seen a field hit this state. And maybe having a few decades of experience with a field makes a radical change in the rate of progress, and the specifics of that, easier to see. Do you have the sense that general AI is now within a decade or so or less? The variety of glass half full, vs. glass half empty judgements around every advance is interesting to me. I find different people have very different sensitivities to constant steps vs. the first and second derivatives of progress they are signaling. 2022 saw more algorithmic progress in poetry, essay writing, graphic art, debate, joke construction, etc, etc, etc. than a human could make in … well a lot more than a year. That’s a new red line we have now crossed. That’s faster than human process. Much faster.
- YeGoblynQueenne 4y ago
  Thanks for the clarifications. >> Do you have the sense that general AI is now within a decade or so or less? Not at all. My impression is that nobody is really working on creating general AI, and that if you take the field of machine learning research as a whole, if the purpose ever was to create general AI (which it never was), the field would be currently running backwards, far away from any kind of "intelligence", or any kind of generality at all. It is beyond clear to me that any sort of generalisation cannot be achieved by neural nets trained with gradient optimisation. That's because gradient optimisation will always, always get stuck at local optima. As you seem to know yourself, gradient optimisation has been around for many decades and I reckon there's been quite a few attempts to see if it can avoid getting stuck at local optima. And yet, well, it still gets stuck at local optima. That's probably its strength anyway, although it's also how it is self-limited. So there's only so much you can do when training with gradient optimisation. There is nothing to do about it. There is nobody who can do anything about it. All the billions of research, all the hordes of researchers, cannot get Humpty Dumpty unstuck from local optima. And so there's not a chance of a snowball in hell that general AI is going to be anywhere near where we get to in a hundred years from now, with current approaches. >> I haven’t ever seen a field hit this state. Well, I agree, but I disagree fundamentally. For me, the state of the field is deplorable! "The field" of machine learning research, that is. That field has abandoned every semblance of doing science and it has become a complete circus, where all that matters is what is the next big company that's paying billions for researchers who know how to train a neural net. Just look at the papers. Thousands of papers are placed on arxiv every month. Hundreds are even published. The vast, vast majority of them do nothing useful at all, and just claim to have moved the state of the art forward a couple of percentage points. They beat a benchmark, brag about it, and go home happy that they will be remembered... for a couple days at least, until someone else with no ideas breaks their record. It's a glorified Kaggle competition, and a free for all, but it achieves nothing. There are no theoretical results in any of the papers, except those of the very few people left at the top of the mess, who would actually know science if it bit them in the leg. There is nothing but noise. And if anyone comes up with a novel idea, in all this mess, that novel idea will be buried under a ton of dreary, miserable dross. Or, to cut this short, what you are seeing is supply and demand: big tech corps supply millions, graduate students run towards them with open arms to get the millions. Progress? What, in that situation? Not really.