3 ms·

While I find this story a fun and intelligent allegory for the issue of ASIs and a fun thought experiment, I find all the doomsaying around ai safety research k

While I find this story a fun and intelligent allegory for the issue of ASIs and a fun thought experiment, I find all the doomsaying around ai safety research kind of tiring. AI safety research is an important and worthwhile field without us needing to constantly be worrying about a far off doomsday that may never come. Yes ASIs are an existential threat and yes once they are out, they are out, and there's no stopping them nor turning them off or anything so we better get them right the first time, but like we don't seem to be anywhere close to ASIs. I have my doubts we will even achieve AGI in this century let alone ASI which I'm not even sure are possible for us non-super-intellegences to create. It seems obvious to me that we are still several major breakthroughs away from AGIs and if these are anything like the deeplearning breakthrough we will probably have a few AI winters between now and then, we have plenty of time to see this supposed doomsday scenario coming and preparing for it. And again this is not to say that ai safety research isn't important. The alignment problem is very much an important and immediate issue right now for Narrow AI let alone for AGI. But seriously the ai isn't going to go from Narrow to running a thousand loops around the whole of humanity in 2 seconds flat, no need for all this doomsaying.

lucubratory 3y ago
What specific demonstrations of capability by AI would convince you that your proposed timeline for reaching AGI isn't correct, and that it will happen significantly sooner? If it helps to think about it, there's a good DeepMind paper on a relatively concrete scaling system on the path to AGI: https://arxiv.org/abs/2311.02462 https://arxiv.org/abs/2311.02462
- xcode42 3y ago
  Thank you for the paper, this seems awesome, I'll have a look later. Deepmind pretty consistently comes up with interesting ai research results, so this should be pretty neat at the very least. Regarding what would convince me that AGI is around the corner, I guess I should start by defining what I'm thinking of when I say AGI, because to be fair AGI is pretty vague. I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing. My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI. Sorry for the long post hope this helped explain my point of view on the issue.
  - yowlingcat 3y ago
    I find this model pretty fascinating. Said another way, there are two aspects necessary for AGI as you see it: - Independence + continual improvement (it's always either doing something useful whether you tell it to or not) - Reliable, prompt and accurate learning of new material The first one is interesting because it's theoretically possible to try right now (just put an LLM scanning in a background job), but clearly not trivial. But I think the second issue you raise is the more fundamental one. There have been a couple papers I've seen that have demonstrated that most major LLMs struggle with recalling things in a way that makes use of equivalence and substitution -- for example, if I tell GPT that X is Y's son, then if it will know that if I ask who is the father of Y, it understands the answer is X. If it cannot show basic competence in equivalence and substitution, then it begins to break through the 4th wall of a "sentient entity" and phenomenologically reveal its true colors as a stochastic parrot. It's fascinating. There are conceptual issues, as you mention, and they're not just theoretical, but they're very noticeable (for now). Here's the question I would ask you -- do you think even if we're no where near AGI, the world has fundamentally changed since the mass distribution of LLMs? I have felt a noticeable shift as the technology's adoption is beginning to alter how society conceives of and digests media. It feels like a discrete new evolutionary state of the internet. Social media fabric feels like it's in a new phase compared to even a year ago as new weapons of mass production have been released.
    - xcode42 3y ago
      Regarding the first one, it's actually a surprisingly difficult problem. Sure we can try right now but deep learning systems currently have a lot of issues with continuous learning, if you just do it the naive way they keep forgetting all they learned to get better performance on the most recent examples. But yeah the second one definitely seems like the harder of the two. The building blocks of intelligence and the mind are indeed endless fascinating, I spend way too much time thinking about this stuff. Oh no I agree, the current AI systems have already caused notable changes in society and even if the development of even more powerful AI stagnates here(for now) we are bound to continue to see societal changes for some time as this is all quite new and society is still adapting and figuring out the "right way" to handle all this. In particular the impact it has had on artists of all kinds is widespread and I'll be curious to see how it pans out and how we handle it as a society.

lucubratory 3y ago

What specific demonstrations of capability by AI would convince you that your proposed timeline for reaching AGI isn't correct, and that it will happen significantly sooner? If it helps to think about it, there's a good DeepMind paper on a relatively concrete scaling system on the path to AGI: https://arxiv.org/abs/2311.02462 https://arxiv.org/abs/2311.02462

xcode42 3y ago
Thank you for the paper, this seems awesome, I'll have a look later. Deepmind pretty consistently comes up with interesting ai research results, so this should be pretty neat at the very least. Regarding what would convince me that AGI is around the corner, I guess I should start by defining what I'm thinking of when I say AGI, because to be fair AGI is pretty vague. I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing. My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI. Sorry for the long post hope this helped explain my point of view on the issue.
- yowlingcat 3y ago
  I find this model pretty fascinating. Said another way, there are two aspects necessary for AGI as you see it: - Independence + continual improvement (it's always either doing something useful whether you tell it to or not) - Reliable, prompt and accurate learning of new material The first one is interesting because it's theoretically possible to try right now (just put an LLM scanning in a background job), but clearly not trivial. But I think the second issue you raise is the more fundamental one. There have been a couple papers I've seen that have demonstrated that most major LLMs struggle with recalling things in a way that makes use of equivalence and substitution -- for example, if I tell GPT that X is Y's son, then if it will know that if I ask who is the father of Y, it understands the answer is X. If it cannot show basic competence in equivalence and substitution, then it begins to break through the 4th wall of a "sentient entity" and phenomenologically reveal its true colors as a stochastic parrot. It's fascinating. There are conceptual issues, as you mention, and they're not just theoretical, but they're very noticeable (for now). Here's the question I would ask you -- do you think even if we're no where near AGI, the world has fundamentally changed since the mass distribution of LLMs? I have felt a noticeable shift as the technology's adoption is beginning to alter how society conceives of and digests media. It feels like a discrete new evolutionary state of the internet. Social media fabric feels like it's in a new phase compared to even a year ago as new weapons of mass production have been released.
  - xcode42 3y ago
    Regarding the first one, it's actually a surprisingly difficult problem. Sure we can try right now but deep learning systems currently have a lot of issues with continuous learning, if you just do it the naive way they keep forgetting all they learned to get better performance on the most recent examples. But yeah the second one definitely seems like the harder of the two. The building blocks of intelligence and the mind are indeed endless fascinating, I spend way too much time thinking about this stuff. Oh no I agree, the current AI systems have already caused notable changes in society and even if the development of even more powerful AI stagnates here(for now) we are bound to continue to see societal changes for some time as this is all quite new and society is still adapting and figuring out the "right way" to handle all this. In particular the impact it has had on artists of all kinds is widespread and I'll be curious to see how it pans out and how we handle it as a society.