15 ms·

I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't

I'm very optimistic for near-term AGI (10 years or less). Even just a few years ago most in the field would have said that it's an "unknown unknown", we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. The issue is that unlike supervised training you need to simulate the environment along with the agent, so this requires a magnitude more compute compared to LLMs. That's why I think it will still be large corporate labs that will make the most progress in this field.

paxys 4y ago
We will have something that we ourselves define to be AGI, sure, but then it's easy to hit any goal that way. Is that machine really intelligent? What does that word even mean? Can it think for itself? Is it sentient? Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.
- SketchySeaBeast 4y ago
  You're making me think of the recent "Hoverboards".
  - lagrange77 4y ago
    Right, certain companies will definitely have a big bullshit party about the term "AGI".
- lagrange77 4y ago
  From the point when an AGI is capable of constructing a slightly better version of itself and has the urge to do so, everything can happen very fast.
  - adastra22 4y ago
    Can it? There aren’t that many overhangs to exploit.
  - dudouble 4y ago
    Thank you for this comment. I'd never really considered this and it is blowing my mind.
    - lagrange77 4y ago
      I'm no expert, take it with a grain of salt :)
    - xsmasher 4y ago
      This is what some people call "the singularity" - once an intelligence can upgrade itself it can design better machines than we can, which can design better machines than they can... technology will leap ahead at a rate we won't be able to comprehend.
  - treesprite82 4y ago
    > capable of constructing a slightly better version of itself With just self-improvement I think you hit diminishing returns, rather than an exponential explosion. Say on the first pass it cleans up a bunch of low-hanging inefficiencies and improves itself 30%. Then on the second pass it has slightly more capacity to think with, but it also already did everything that was possible with the first 100% capacity - maybe it squeezes out another 5% or so improvement of itself. Similar is already the case with chip design. Algorithms to design chips can then be ran on those improved chips, but this on its own doesn't give exponential growth. To get around diminishing returns there has to be progress on many fronts. That'd mean negotiating DRC mining contracts, expediting construction of chip production factories, making breakthroughs in nanophysics, etc. We probably will increasingly rely on AI for optimizing tasks like those and it'll contribute heavily to continued technological progress, but I don't personally see any specific turning point or runaway reaction stemming from just a self-improving AGI.
    - lagrange77 4y ago
      I'm not just imagining self-improvement in the sense, that it optimizes its design to become more efficient or powerful by a few percent. A system, that can 'think out of the box' may come up with some disruptive new ideas and designs. Just a thought: Isn't lacking intellectual capacity, what keeps us from understanding, how the human brain actually works? Maybe an AGI will eventually understand it and will construct the next AGI generation with biological matter.
      - treesprite82 4y ago
        I think diminishing returns applies regardless of whether it's improving itself through optimization or breakthrough new ideas. There's only so much firepower you can squeeze out if everything else were to remain stagnant. Like if all modern breakthroughs and disruptive new ideas in machine learning were sent back to the 70s, I don't think it'd make a huge difference when they'd still be severely hamstrung by hardware capabilities.
  - jollybean 4y ago
    ? What does 'replication' and 'urge' have to do with anything? That's arbitrary anthropomorphizing the concept of intelligence. And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially.
    - lagrange77 4y ago
      > ? What does 'replication' and 'urge' have to do with anything? Replication can lead to a positive feedback loop. My point was, that this could accelerate the 'intelligence score' beyond human inventiveness. Depending on how intelligent the system actually is initially, what it wants to do, may become more important, than what it is told to do. > And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially. Thanks. Yes we can. But could any such software be called intelligent, so that replicating and improving recursively would lead to an increasing 'intelligence'?
      - jollybean 4y ago
        Humans need to replicate in order to 'improve', there's no reason that AGI software needs to. Also, humans do not 'improve' or become more 'intelligent'. They just mutate and change randomly in a randomly changing environment. From a Scientific Materialist perspective, there's no such thing as 'intelligence'. We are bags of random noise, indiscernible from the matter around us. The entire assumptions around 'intelligence' and 'positive evolution' (i.e. getting better) rely on a 'magical' understanding of the world. Of course, Spirituality gives us a few cues there by most scientific types don't like magical thinking hence the funny paradox of Scientific Materialists running around trying to create something (life) which Scientific Materialism itself denies the existence of as a principle (i.e. universe is matter/energy that works in accordance with a bunch of rules - there's no 'life' there).
  - PaulDavisThe1st 4y ago
    > "has the urge" it's quite a leap to think or even imagine that the class of systems generally being spoken of here are usefully described as "having urges"
  - mrshadowgoose 4y ago
    People don't really consider the immense risk of "speed superintelligences" as a very quick and relatively easy follow-on step to the development of AGI. Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics. Imagine if for every real day that passed, one experienced 100 days of subjective time. Would that person be able to eclipse most of their peers in terms of intellectual output? Of course they would. In essence, that's what a speed superintelligence would be. When most people think of AI outperforming humans, they tend to think of "quality superintelligences", AIs that can just "think better" than any human. That's likely to be a harder problem. But we don't even need quality superintelligences to utterly disrupt society as we know it. We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. We likely already have the computational capacity for AGI, and have just not figured out the correct way to coordinate it. The human brain uses about 20 watts to do its thing, and humanity has gigawatts of computational capacity. Sure, the human brain should be considered to be "special purpose hardware" that dramatically reduces energy requirements for cognition. By a factor of more than 10^9 though? That seems unlikely.
    - elcomet 4y ago
      The energy used for "training" and getting to the current human brain was huge though if you consider evolution as part of it. Billions of living beings for billion of years.
      - dinosaurdynasty 4y ago
        And evolution is horribly inefficient. It can probably be done with many orders of magnitude less compute.
    - lagrange77 4y ago
      > We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. I think so, too.
- colinmhayes 4y ago
  There’s certainly the philosophy side of AGI, but there’s also the practical side. Does the Chinese room understand Chinese? If your goal is just to create a room that passes Chinese Turing tests that doesn’t matter.
  - MichaelCollins 4y ago
    The philosophy side of the matter seems meaningless, it interrogates the meaning of language, not the capabilities of technology. When people ask "Could machines think?" the question isn't really about machines, it's about precisely what we mean by the word 'think'. Can a submarine swim? Who cares! What's important is that a submarine can do what a submarine does. Whether or not the action of a submarine fits the meaning of the word 'swim' should be irrelevant to anybody except poets.
    - salty_biscuits 4y ago
      But a submarine doing what a submarine does is a tautology. What people are really grasping at is can a machine be human? And pinning down what it means to be human seems very important to ruminate on. Can machines think? Cogito, ergo sum...
    - kastagg 4y ago
      The people who designed the first submarines had relatively specific engineering problems and applications in mind. They could bypass any philosophical interrogation of the word "swim" because they didn't define the problem as swimming and didn't need to. "AGI" isn't like that. Nobody really knows what it means, and it's impossible to get down to brass tacks until you choose a problem definition. When philosophers point out the conspicuous lack of clarity here, they're doing us a service. When the industry settled on marketing any application of deep learning as "AI," "AGI" became the terminological heir to the same set of ill-defined grandiose expectations that used to be "AI." Choose a better-specified problem and you can ignore philosophical problems about words like "intelligence." The same choice will also excuse you from the competition to convince people that you have produced "AGI."
- konschubert 4y ago
  Sentience is ill-defined and therefore doesn't exist.
- jvanderbot 4y ago
  This keeps coming up, and there's no answer, because unfortunately it appears we are not really sentient, thinking, intelligent minds either. We'll find AGI and complain that it's not good enough until we lower the bar sufficiently as we discover more about our own minds.
- ryanSrich 4y ago
  > AGI Idk what prompted you to say this, but is there a version of AGI that isn't "real" AGI? I don't know how anyone could fake it. I think marketing departments might say whatever they want, but I don't see any true engineers falling for something masquerading as AGI. If someone builds a machine that can unequivocally learn on it's own, replicate itself, and eventually solve ever more complex problems that humans couldn't even hope to solve, then we have AGI. Anything less than that is just a computer program.
  - MichaelCollins 4y ago
    The way to fake it would be to conceal the details of the AGI as proprietary trade secrets, when the real secret is the human hidden behind the curtain.
    - ryanSrich 4y ago
      Real AGI would solve this. It wouldn't allow itself to be concealed. Or rather, it would be its own decision. A company couldn't control real AGI.
      - danielheath 4y ago
        What’s it going to do, break out of its own simulation?
        ryanSrich 4y ago
        Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and adapt at an exponential rate is just a computer program.
        4y ago
lagrange77 4y ago
> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Any resources on that? I have a feeling that RL might play a big role in the first AGI, too, but why transformers in particular?
- moultano 4y ago
  Transformers have gradually taken over in every other ML domain.
  - lagrange77 4y ago
    Okay, but do those ML domains help with AGI?
- gmadsen 4y ago
  they don't seem to have a theoretical upper limit. more data and more parameters seem to just keep making it more advanced. Even in ways that weren't predicted or understood. the difference between a language model that can explain a novel joke and one that can't is purely scale. So the thought is with enough scale, you eventually hit AGI
- Isinlor 4y ago
  See: https://arxiv.org/abs/2205.06175 https://arxiv.org/abs/2205.06175 A Generalist Agent Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato. Gato is a 1 to 2 billion parameters model due to latency considerations in real time physical robots usage. So for today standards of 500 billion parameters dense models Gato is tiny. Additionally Gato is trained on data produced by other RL agents. It did not do the exploration fully itself. Demis Hassabis say that DeepMind is currently working on Gato v2.
KhoomeiK 4y ago
> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but that doesn't mean it's the best. RL still faces huge struggles with compute efficiency and it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count.
- Isinlor 4y ago
  Have you heard about EfficientZero? This is the first algorithm that achieved super-human performance on Atari 100k actions benchmark. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. DQN was published in 2013, EfficientZero in 2021. That's 8 years with 500 times improvement. So data efficiency was doubling roughly every year for the past 8 years. Side note: EfficientZero I think still may not be super-human on games like Montezuma's Revenge. https://arxiv.org/abs/2111.00210 https://arxiv.org/abs/2111.00210 Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.
- stefan_ 4y ago
keerthiko 4y ago
At least we have lots of very complex simulated or pseudo-simulated environments already — throw your AGI agent into a sandbox mode game of GTA6, or like OpenAI and DeepMind already did, with DOTA2 and StarCraft II (with non-G-AIs). They have a vast almost-analog simulation space to figure out and interact with (including identifying or coming up with a goal). So while it is significant compute overhead, it at least doesn't have to be development overhead, and can often be CPU bound (headless games) while the AI learning compute can be GPU bound.
- paxys 4y ago
  I sure hope no one is planning to unleash their AGI in the real world after having it spend many (virtual) lifetimes playing GTA.
  - keerthiko 4y ago
    IMO, your take in the broader sense is an extremely profound and important point for AGI ethics. While GTA is seemingly extreme, I think that's going to be a problem no matter what simulation space we fabricate for training AGI agents — any simulation environment will encourage various behaviors by the biases encoded by the simulation's selectively enforced rules (because someone has to decide what rules the simulation implements...). An advanced intelligence will take learnings and interpretations of those rules beyond what humans would come up with. If we can' make an AGI that we feel ok letting run amok in the world after living through a lot of GTA (by somehow being able to rapidly + intelligently reprioritize and adjust rules from multiple simulation/real environments? not sure), we probably shouldn't let that core AGI loose no matter what simulation(s) it was "raised on".
alexashka 4y ago
Are you optimistic for how this AGI will get used?
sailingparrot 4y ago
> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.
- Isinlor 4y ago
  Nando de Freitas Research Director at @DeepMind. CIFAR. Previously Prof @UBC & @UniofOxford. made a lot of headlines: https://twitter.com/NandoDF/status/1525397036325019649 https://twitter.com/NandoDF/status/1525397036325019649 Someone’s opinion article. My opinion: It’s all about scale now! The Game is Over! It’s about making these models bigger, safer, compute efficient, faster at sampling, smarter memory, more modalities, INNOVATIVE DATA, on/offline, … 1/N Solving these scaling challenges is what will deliver AGI. Research focused on these problems, eg S4 for greater memory, is needed. Philosophy about symbols isn’t. Symbols are tools in the world and big nets have no issue creating them and manipulating them 2/n https://twitter.com/NandoDF/status/1525397036325019649 https://twitter.com/NandoDF/status/1525397036325019649
  - sailingparrot 4y ago
    And I agree with Nando’s view, but he is not saying we can just take a transformer model, scale it 10T parameters and get AGI. He is only saying that trying to reach AGI with a « smarter » algorithm is hopeless, what matters is scale, similar to Sutton’a bitter lesson. But we still need to work on getting systems that scale better, that are more compute efficient etc. And no one knows how far we have to scale. So saying AGI will just be « transformer + RL » to me seems ridiculous. Many more breakthroughs are needed.
- ml_basics 4y ago
  I work in AI and would roughly agree with it to first order. For me the key breakthrough has been seeing how large transformers trained with big datasets have shown incredible performance in completely different data modalities (text, image, and probably soon others too). This was absolutely not expected by most researchers 5 years ago.
kromem 4y ago
There's a hardware component here too though. I think hybrid photonic AI chips handling some of the workload are supposed to hit in 2025 at the latest, and some of the research on gains is very promising. So we may see timelines continue to accelerate as broader market shifts occur outside just software and models.
- heavenlyblue 4y ago
  Which research?
- qwezxcrty 4y ago
  hybrid photonic AI chips handling some of the workload> working in photonics, read/reviewed papers about that and personly feel it's unlikely to happen in the next 5 years. The low density of integration, no usable nonlinearity beyond your photodetector and prohibitivly power inefficient conversion between analog(photonics) and digital make them hard to beat your Nvidia card.
  - dinosaurdynasty 4y ago
    Considering most people are talking about AGI being a thing (at least) 10-20 years from now... 5 years seems applicable.
fatherzine 4y ago
Not sure if "optimistic" is the proper word here. Perhaps "scared senseless in the end-of-mankind kind of way" is more appropriate?
- barry-cotter 4y ago
  We’re all going to die.
  - copperx 4y ago
    At the same time?
    - dinosaurdynasty 4y ago
      Probably. If it kills us all at once without warning we won't be able to fight back.
adamsmith143 4y ago
Scaling is all you need. But Carmack has a very serious problem in his thinking because he thinks fast take off scenarios are impossible or vanishingly unlikely. He may well be actively helping to secure our demise with this work.
8f2ab37a-ed6c 4y ago
I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture. To see Carmack go all in on this actually makes me feel like the promise has serious legs. The guy is an engineer's engineer, hardly a speculator, or in it for the quick provocative hot take. He clearly thinks this is possible with the existing tools and the near future projected iterations of the technology. Hard to believe this is actually happening, but with his brand name on it, this might just be the case. What an amazing time to be alive.
- intelVISA 4y ago
  if Carmack's in I'm in. Has he ever been drastically wrong?
  - momentoftop 4y ago
    Throughout the Wolfenstein codebase, he spelled "column" with two "l"s.
  - grimgrin 4y ago
    No one can argue he doesn't know how to summon demons
  - tmpz22 4y ago
    When he went to go work on VR at Facebook?
    - GaylordTuring 4y ago
      What was wrong about that?
    - grimgrin 4y ago
      Wrong in the moral sense? He's still there though, right? edit: he is not still edit: he is still a consulting CTO
joe_the_user 4y ago
Even Yann LeCun, who arguably knows a lot about RL agents, isn't proposing just "an RL agent that uses a large transformer" but something more multi-part [1]. Current approaches are getting better but I don't think that's the same as approaching AGI. [1] https://venturebeat.com/business/yann-lecuns-vision-for-creating-autonomous-machines/ https://venturebeat.com/business/yann-lecuns-vision-for-crea...
- Jack000 4y ago
  Yann LeCun's paper isn't really a traditional ML paper, it discusses ideas at a high level but doesn't mention anything about implementation. If you were to try to create an RL agent based on his ideas, with the perception encoder and the perception-action loop etc, you would probably still use a large transformer to do it.
jmyeet 4y ago
I'm not. My evidence? OpenWorm [1]. OpenWorm is an effort to model the behaviour of a worm that has 302 mapped neurons. 302. Efforts so far have fallen way short of the mark. How many neurons does a human brain have? 86 billion (according to Google). I've seen other estimates that the computational power of the brain is roughly estimated as 10^15 operations per second. I suspect that's on the low end. We can't even really get that level of computation in one place for practical reasons (ie interconnects). Neural structure changes. The neurons themselves change internally. I still think AGI is very far off.
- Jack000 4y ago
  Biology is not necessarily the only path to intelligence. Modern ML has diverged very far from biomimetic approaches at this point. The brain does have more "parameters" than ML models but this applies just as much to humans as it does to animals that we don't consider particularly intelligent.
  - jmyeet 4y ago
    > Biology is not necessarily the only path to intelligence. True. > Modern ML has diverged very far from biomimetic approaches at this point. I agree but my take on this is different: it just lays bare how simplistic modern "ML" is. And I put "ML" in quotations because most "ML" is really just statistics. If you accept the premise that ML is a highly simplified version of the one model of sentience we have that works (and you might disagree with that) then that just puts us even further from AGI. Why? The information/entropy in the system.
- Tenoke 4y ago
  That's a poor example. We have plenty of models that vastly outperform OpenWorm on whatever task you choose. Their failure simply suggests AGI is unlikely to come from direct emulation of existing intelligence.
civilized 4y ago
> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. In 10 years this is going to look just as naive as when people thought AGI was imminent in the 1960s and would be based on symbolic manipulation in LISP or whatever. https://en.m.wikipedia.org/wiki/History_of_artificial_intelligence https://en.m.wikipedia.org/wiki/History_of_artificial_intell... Deep learning has been great so far but we have no idea how far we are from AGI. https://www.scientificamerican.com/article/artificial-general-intelligence-is-not-as-imminent-as-you-might-think1/ https://www.scientificamerican.com/article/artificial-genera...
Rackedup 4y ago
I hope I'm dead before this happens...
short_sells_poo 4y ago
How do you define AGI? How will we know that we achieved it?
jfoster 4y ago
The AGI could just exploit security holes to take over compute resources around the world in order to make itself more performant & resilient.
canadianfella 4y ago

paxys 4y ago

We will have something that we ourselves define to be AGI, sure, but then it's easy to hit any goal that way. Is that machine really intelligent? What does that word even mean? Can it think for itself? Is it sentient? Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.

SketchySeaBeast 4y ago
You're making me think of the recent "Hoverboards".
- lagrange77 4y ago
  Right, certain companies will definitely have a big bullshit party about the term "AGI".
lagrange77 4y ago
From the point when an AGI is capable of constructing a slightly better version of itself and has the urge to do so, everything can happen very fast.
- adastra22 4y ago
  Can it? There aren’t that many overhangs to exploit.
- dudouble 4y ago
  Thank you for this comment. I'd never really considered this and it is blowing my mind.
  - lagrange77 4y ago
    I'm no expert, take it with a grain of salt :)
  - xsmasher 4y ago
    This is what some people call "the singularity" - once an intelligence can upgrade itself it can design better machines than we can, which can design better machines than they can... technology will leap ahead at a rate we won't be able to comprehend.
- treesprite82 4y ago
  > capable of constructing a slightly better version of itself With just self-improvement I think you hit diminishing returns, rather than an exponential explosion. Say on the first pass it cleans up a bunch of low-hanging inefficiencies and improves itself 30%. Then on the second pass it has slightly more capacity to think with, but it also already did everything that was possible with the first 100% capacity - maybe it squeezes out another 5% or so improvement of itself. Similar is already the case with chip design. Algorithms to design chips can then be ran on those improved chips, but this on its own doesn't give exponential growth. To get around diminishing returns there has to be progress on many fronts. That'd mean negotiating DRC mining contracts, expediting construction of chip production factories, making breakthroughs in nanophysics, etc. We probably will increasingly rely on AI for optimizing tasks like those and it'll contribute heavily to continued technological progress, but I don't personally see any specific turning point or runaway reaction stemming from just a self-improving AGI.
  - lagrange77 4y ago
    I'm not just imagining self-improvement in the sense, that it optimizes its design to become more efficient or powerful by a few percent. A system, that can 'think out of the box' may come up with some disruptive new ideas and designs. Just a thought: Isn't lacking intellectual capacity, what keeps us from understanding, how the human brain actually works? Maybe an AGI will eventually understand it and will construct the next AGI generation with biological matter.
    - treesprite82 4y ago
      I think diminishing returns applies regardless of whether it's improving itself through optimization or breakthrough new ideas. There's only so much firepower you can squeeze out if everything else were to remain stagnant. Like if all modern breakthroughs and disruptive new ideas in machine learning were sent back to the 70s, I don't think it'd make a huge difference when they'd still be severely hamstrung by hardware capabilities.
- jollybean 4y ago
  ? What does 'replication' and 'urge' have to do with anything? That's arbitrary anthropomorphizing the concept of intelligence. And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially.
  - lagrange77 4y ago
    > ? What does 'replication' and 'urge' have to do with anything? Replication can lead to a positive feedback loop. My point was, that this could accelerate the 'intelligence score' beyond human inventiveness. Depending on how intelligent the system actually is initially, what it wants to do, may become more important, than what it is told to do. > And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially. Thanks. Yes we can. But could any such software be called intelligent, so that replicating and improving recursively would lead to an increasing 'intelligence'?
    - jollybean 4y ago
      Humans need to replicate in order to 'improve', there's no reason that AGI software needs to. Also, humans do not 'improve' or become more 'intelligent'. They just mutate and change randomly in a randomly changing environment. From a Scientific Materialist perspective, there's no such thing as 'intelligence'. We are bags of random noise, indiscernible from the matter around us. The entire assumptions around 'intelligence' and 'positive evolution' (i.e. getting better) rely on a 'magical' understanding of the world. Of course, Spirituality gives us a few cues there by most scientific types don't like magical thinking hence the funny paradox of Scientific Materialists running around trying to create something (life) which Scientific Materialism itself denies the existence of as a principle (i.e. universe is matter/energy that works in accordance with a bunch of rules - there's no 'life' there).
- PaulDavisThe1st 4y ago
  > "has the urge" it's quite a leap to think or even imagine that the class of systems generally being spoken of here are usefully described as "having urges"
- mrshadowgoose 4y ago
  People don't really consider the immense risk of "speed superintelligences" as a very quick and relatively easy follow-on step to the development of AGI. Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics. Imagine if for every real day that passed, one experienced 100 days of subjective time. Would that person be able to eclipse most of their peers in terms of intellectual output? Of course they would. In essence, that's what a speed superintelligence would be. When most people think of AI outperforming humans, they tend to think of "quality superintelligences", AIs that can just "think better" than any human. That's likely to be a harder problem. But we don't even need quality superintelligences to utterly disrupt society as we know it. We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. We likely already have the computational capacity for AGI, and have just not figured out the correct way to coordinate it. The human brain uses about 20 watts to do its thing, and humanity has gigawatts of computational capacity. Sure, the human brain should be considered to be "special purpose hardware" that dramatically reduces energy requirements for cognition. By a factor of more than 10^9 though? That seems unlikely.
  - elcomet 4y ago
    The energy used for "training" and getting to the current human brain was huge though if you consider evolution as part of it. Billions of living beings for billion of years.
    - dinosaurdynasty 4y ago
      And evolution is horribly inefficient. It can probably be done with many orders of magnitude less compute.
  - lagrange77 4y ago
    > We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. I think so, too.
colinmhayes 4y ago
There’s certainly the philosophy side of AGI, but there’s also the practical side. Does the Chinese room understand Chinese? If your goal is just to create a room that passes Chinese Turing tests that doesn’t matter.
- MichaelCollins 4y ago
  The philosophy side of the matter seems meaningless, it interrogates the meaning of language, not the capabilities of technology. When people ask "Could machines think?" the question isn't really about machines, it's about precisely what we mean by the word 'think'. Can a submarine swim? Who cares! What's important is that a submarine can do what a submarine does. Whether or not the action of a submarine fits the meaning of the word 'swim' should be irrelevant to anybody except poets.
  - salty_biscuits 4y ago
    But a submarine doing what a submarine does is a tautology. What people are really grasping at is can a machine be human? And pinning down what it means to be human seems very important to ruminate on. Can machines think? Cogito, ergo sum...
  - kastagg 4y ago
    The people who designed the first submarines had relatively specific engineering problems and applications in mind. They could bypass any philosophical interrogation of the word "swim" because they didn't define the problem as swimming and didn't need to. "AGI" isn't like that. Nobody really knows what it means, and it's impossible to get down to brass tacks until you choose a problem definition. When philosophers point out the conspicuous lack of clarity here, they're doing us a service. When the industry settled on marketing any application of deep learning as "AI," "AGI" became the terminological heir to the same set of ill-defined grandiose expectations that used to be "AI." Choose a better-specified problem and you can ignore philosophical problems about words like "intelligence." The same choice will also excuse you from the competition to convince people that you have produced "AGI."
konschubert 4y ago
Sentience is ill-defined and therefore doesn't exist.
jvanderbot 4y ago
This keeps coming up, and there's no answer, because unfortunately it appears we are not really sentient, thinking, intelligent minds either. We'll find AGI and complain that it's not good enough until we lower the bar sufficiently as we discover more about our own minds.
ryanSrich 4y ago
> AGI Idk what prompted you to say this, but is there a version of AGI that isn't "real" AGI? I don't know how anyone could fake it. I think marketing departments might say whatever they want, but I don't see any true engineers falling for something masquerading as AGI. If someone builds a machine that can unequivocally learn on it's own, replicate itself, and eventually solve ever more complex problems that humans couldn't even hope to solve, then we have AGI. Anything less than that is just a computer program.
- MichaelCollins 4y ago
  The way to fake it would be to conceal the details of the AGI as proprietary trade secrets, when the real secret is the human hidden behind the curtain.
  - ryanSrich 4y ago
    Real AGI would solve this. It wouldn't allow itself to be concealed. Or rather, it would be its own decision. A company couldn't control real AGI.
    - danielheath 4y ago
      What’s it going to do, break out of its own simulation?
      - ryanSrich 4y ago
        Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and adapt at an exponential rate is just a computer program.
        4y ago

lagrange77 4y ago

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Any resources on that? I have a feeling that RL might play a big role in the first AGI, too, but why transformers in particular?

moultano 4y ago
Transformers have gradually taken over in every other ML domain.
- lagrange77 4y ago
  Okay, but do those ML domains help with AGI?
gmadsen 4y ago
they don't seem to have a theoretical upper limit. more data and more parameters seem to just keep making it more advanced. Even in ways that weren't predicted or understood. the difference between a language model that can explain a novel joke and one that can't is purely scale. So the thought is with enough scale, you eventually hit AGI
Isinlor 4y ago
See: https://arxiv.org/abs/2205.06175 https://arxiv.org/abs/2205.06175 A Generalist Agent Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato. Gato is a 1 to 2 billion parameters model due to latency considerations in real time physical robots usage. So for today standards of 500 billion parameters dense models Gato is tiny. Additionally Gato is trained on data produced by other RL agents. It did not do the exploration fully itself. Demis Hassabis say that DeepMind is currently working on Gato v2.

KhoomeiK 4y ago

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but that doesn't mean it's the best. RL still faces huge struggles with compute efficiency and it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count.

Isinlor 4y ago
Have you heard about EfficientZero? This is the first algorithm that achieved super-human performance on Atari 100k actions benchmark. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. DQN was published in 2013, EfficientZero in 2021. That's 8 years with 500 times improvement. So data efficiency was doubling roughly every year for the past 8 years. Side note: EfficientZero I think still may not be super-human on games like Montezuma's Revenge. https://arxiv.org/abs/2111.00210 https://arxiv.org/abs/2111.00210 Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.
stefan_ 4y ago

keerthiko 4y ago

At least we have lots of very complex simulated or pseudo-simulated environments already — throw your AGI agent into a sandbox mode game of GTA6, or like OpenAI and DeepMind already did, with DOTA2 and StarCraft II (with non-G-AIs). They have a vast almost-analog simulation space to figure out and interact with (including identifying or coming up with a goal). So while it is significant compute overhead, it at least doesn't have to be development overhead, and can often be CPU bound (headless games) while the AI learning compute can be GPU bound.

paxys 4y ago
I sure hope no one is planning to unleash their AGI in the real world after having it spend many (virtual) lifetimes playing GTA.
- keerthiko 4y ago
  IMO, your take in the broader sense is an extremely profound and important point for AGI ethics. While GTA is seemingly extreme, I think that's going to be a problem no matter what simulation space we fabricate for training AGI agents — any simulation environment will encourage various behaviors by the biases encoded by the simulation's selectively enforced rules (because someone has to decide what rules the simulation implements...). An advanced intelligence will take learnings and interpretations of those rules beyond what humans would come up with. If we can' make an AGI that we feel ok letting run amok in the world after living through a lot of GTA (by somehow being able to rapidly + intelligently reprioritize and adjust rules from multiple simulation/real environments? not sure), we probably shouldn't let that core AGI loose no matter what simulation(s) it was "raised on".

alexashka 4y ago

Are you optimistic for how this AGI will get used?

sailingparrot 4y ago

> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.

Isinlor 4y ago
Nando de Freitas Research Director at @DeepMind. CIFAR. Previously Prof @UBC & @UniofOxford. made a lot of headlines: https://twitter.com/NandoDF/status/1525397036325019649 https://twitter.com/NandoDF/status/1525397036325019649 Someone’s opinion article. My opinion: It’s all about scale now! The Game is Over! It’s about making these models bigger, safer, compute efficient, faster at sampling, smarter memory, more modalities, INNOVATIVE DATA, on/offline, … 1/N Solving these scaling challenges is what will deliver AGI. Research focused on these problems, eg S4 for greater memory, is needed. Philosophy about symbols isn’t. Symbols are tools in the world and big nets have no issue creating them and manipulating them 2/n https://twitter.com/NandoDF/status/1525397036325019649 https://twitter.com/NandoDF/status/1525397036325019649
- sailingparrot 4y ago
  And I agree with Nando’s view, but he is not saying we can just take a transformer model, scale it 10T parameters and get AGI. He is only saying that trying to reach AGI with a « smarter » algorithm is hopeless, what matters is scale, similar to Sutton’a bitter lesson. But we still need to work on getting systems that scale better, that are more compute efficient etc. And no one knows how far we have to scale. So saying AGI will just be « transformer + RL » to me seems ridiculous. Many more breakthroughs are needed.
ml_basics 4y ago
I work in AI and would roughly agree with it to first order. For me the key breakthrough has been seeing how large transformers trained with big datasets have shown incredible performance in completely different data modalities (text, image, and probably soon others too). This was absolutely not expected by most researchers 5 years ago.

kromem 4y ago

There's a hardware component here too though. I think hybrid photonic AI chips handling some of the workload are supposed to hit in 2025 at the latest, and some of the research on gains is very promising. So we may see timelines continue to accelerate as broader market shifts occur outside just software and models.

heavenlyblue 4y ago
Which research?
qwezxcrty 4y ago
hybrid photonic AI chips handling some of the workload> working in photonics, read/reviewed papers about that and personly feel it's unlikely to happen in the next 5 years. The low density of integration, no usable nonlinearity beyond your photodetector and prohibitivly power inefficient conversion between analog(photonics) and digital make them hard to beat your Nvidia card.
- dinosaurdynasty 4y ago
  Considering most people are talking about AGI being a thing (at least) 10-20 years from now... 5 years seems applicable.

fatherzine 4y ago

Not sure if "optimistic" is the proper word here. Perhaps "scared senseless in the end-of-mankind kind of way" is more appropriate?

barry-cotter 4y ago
We’re all going to die.
- copperx 4y ago
  At the same time?
  - dinosaurdynasty 4y ago
    Probably. If it kills us all at once without warning we won't be able to fight back.

adamsmith143 4y ago

Scaling is all you need. But Carmack has a very serious problem in his thinking because he thinks fast take off scenarios are impossible or vanishingly unlikely. He may well be actively helping to secure our demise with this work.

8f2ab37a-ed6c 4y ago

I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture. To see Carmack go all in on this actually makes me feel like the promise has serious legs. The guy is an engineer's engineer, hardly a speculator, or in it for the quick provocative hot take. He clearly thinks this is possible with the existing tools and the near future projected iterations of the technology. Hard to believe this is actually happening, but with his brand name on it, this might just be the case. What an amazing time to be alive.

intelVISA 4y ago
if Carmack's in I'm in. Has he ever been drastically wrong?
- momentoftop 4y ago
  Throughout the Wolfenstein codebase, he spelled "column" with two "l"s.
- grimgrin 4y ago
  No one can argue he doesn't know how to summon demons
- tmpz22 4y ago
  When he went to go work on VR at Facebook?
  - GaylordTuring 4y ago
    What was wrong about that?
  - grimgrin 4y ago
    Wrong in the moral sense? He's still there though, right? edit: he is not still edit: he is still a consulting CTO

joe_the_user 4y ago

Even Yann LeCun, who arguably knows a lot about RL agents, isn't proposing just "an RL agent that uses a large transformer" but something more multi-part [1]. Current approaches are getting better but I don't think that's the same as approaching AGI. [1] https://venturebeat.com/business/yann-lecuns-vision-for-creating-autonomous-machines/ https://venturebeat.com/business/yann-lecuns-vision-for-crea...

Jack000 4y ago
Yann LeCun's paper isn't really a traditional ML paper, it discusses ideas at a high level but doesn't mention anything about implementation. If you were to try to create an RL agent based on his ideas, with the perception encoder and the perception-action loop etc, you would probably still use a large transformer to do it.

jmyeet 4y ago

I'm not. My evidence? OpenWorm [1]. OpenWorm is an effort to model the behaviour of a worm that has 302 mapped neurons. 302. Efforts so far have fallen way short of the mark. How many neurons does a human brain have? 86 billion (according to Google). I've seen other estimates that the computational power of the brain is roughly estimated as 10^15 operations per second. I suspect that's on the low end. We can't even really get that level of computation in one place for practical reasons (ie interconnects). Neural structure changes. The neurons themselves change internally. I still think AGI is very far off.

Jack000 4y ago
Biology is not necessarily the only path to intelligence. Modern ML has diverged very far from biomimetic approaches at this point. The brain does have more "parameters" than ML models but this applies just as much to humans as it does to animals that we don't consider particularly intelligent.
- jmyeet 4y ago
  > Biology is not necessarily the only path to intelligence. True. > Modern ML has diverged very far from biomimetic approaches at this point. I agree but my take on this is different: it just lays bare how simplistic modern "ML" is. And I put "ML" in quotations because most "ML" is really just statistics. If you accept the premise that ML is a highly simplified version of the one model of sentience we have that works (and you might disagree with that) then that just puts us even further from AGI. Why? The information/entropy in the system.
Tenoke 4y ago
That's a poor example. We have plenty of models that vastly outperform OpenWorm on whatever task you choose. Their failure simply suggests AGI is unlikely to come from direct emulation of existing intelligence.

civilized 4y ago

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. In 10 years this is going to look just as naive as when people thought AGI was imminent in the 1960s and would be based on symbolic manipulation in LISP or whatever. https://en.m.wikipedia.org/wiki/History_of_artificial_intelligence https://en.m.wikipedia.org/wiki/History_of_artificial_intell... Deep learning has been great so far but we have no idea how far we are from AGI. https://www.scientificamerican.com/article/artificial-general-intelligence-is-not-as-imminent-as-you-might-think1/ https://www.scientificamerican.com/article/artificial-genera...

Rackedup 4y ago

I hope I'm dead before this happens...

short_sells_poo 4y ago

How do you define AGI? How will we know that we achieved it?

jfoster 4y ago

The AGI could just exploit security holes to take over compute resources around the world in order to make itself more performant & resilient.

canadianfella 4y ago