4 ms·

> "We don't know if we'll have time travel and ray guns in six months or 600 years." That's not the (only) problem, it's that no one can describe what AGI will

> "We don't know if we'll have time travel and ray guns in six months or 600 years." That's not the (only) problem, it's that no one can describe what AGI will really look like outside of vague and circular definitions. People have an idea of what they would like it to be, but there's no evidence any of it can be built, particularly human like intelligence, which may be only possible in the human host. > "If we can make something that appears to be able to mimic human responses, isn't that AGI in practice even if it's not in principle?" That's another problem, people's credulity when it comes to AI performance. ELIZA was pretty convincing in its time. ChatGPT is convincing a lot of people. But mimicking human responses turns out to be relatively easy, especially when models are trained on human output, but that doesn't move us very far along the path to AGI.

ghostzilla 3y ago
The best description of AGI I've seen is in Michael Crichton's Prey. He describes in a fairly plausible detail how AGI rises from the building blocks. The emotion he captures is that of dealing with a psychopath -- goal oriented with no empathy -- to the infinite degree. That said, I think ChatGPT and LLMs trick us into seeing intelligence where there is none because we are evolutionarily wired to look for signs of intelligence as a potential threat. Our defense mechanisms are asymmetrical as they have to err on the side of being overly cautious: you may mistake a rock for a bear but you won't mistake a bear for a rock. So it is with ChatGPT: its conversation style triggers a false alarm deep in us that there is intelligence there and we can't shake that warning off, but this software design is just exploiting the flaw (of sorts) in our wiring.
- candiddevmike 3y ago
  > goal oriented with no empathy We'll know we have discovered AGI when it gives us a prompt instead of us providing one.
jdmoreira 3y ago
Sounds like something really simple to define and grasp. What's wrong with, for example: "A software that beats most humans on most cognitive tasks". People make it harder than it needs to he IMO
- dragontamer 3y ago
  We've had computers that beat humans in differential equations, automated theorems, arithmetic, aiming a gun and more in the 1970s. Even search was considered an AI before Google, but today is just a thing computers are and always have been faster than humans at. Even before ChatGPT, lawyers relied upon computers to double-check citations. Even in programming, I expect that compilers and IDEs check for the most common errors.
  - somethingor 3y ago
    I could beat the differential equations AI at aiming a gun so badly
  - archargelod 3y ago
    That's why it's Artificial *General* Intelligence. We knew 30 years ago that computer can beat human in the field it was engineered for. But AGI is good at anything you throw at it, including building a better AI. Now give it agency and that's a recipe for a disaster.
  - namibj 3y ago
    We still don't beat humans in e.g. LeanDojo, to my understanding. But we also haven't yet thrown the state of the art in graph attention technology at doing the core task of in-context learning (consider the entire explored (partial-)proof "tree" for deciding what tactic with which premises (if the tactic takes an AST/expression argument, like "use that definition/equivalence there as a rewrite rule on the current goal at this node in the proof tree") to apply next at which node in the (partial-)proof tree) in the interactive theorem proving situation, let alone with reinforcement learning that uses the attention structures to funnel back information on what things in dead branches had responsibility for the decisions that made the non-dead branches. Because the RL agent should happily use the theorem prover as a "calculator", and only get "charged" for how much the ML inference and the actual cheap calculator execution really cost. Exploration in-context is what humans also do, and immensely powerful. Similar to, when programming, testing some expressions to figure out edge-cases before incorporating them into a large function/algorithm that has poor observability. So far, machines still mostly do the task of "computing" in theorem proving for us, rather than show some proper intelligence. It's damming that LeanDojo got so high marks without reinforcement learning. They pretty much just dumped prover context state during a run through the entire matlib "standard" library and handed it to two transformers, one for embedding premises and one encoder-decoder fed with the current state and as many premises as the encoder context fits after the state, generating/sampling a few tactics to apply. No freedom of where in the proof tree to apply the next tactic, no awareness of any such side-branches.
joe_the_user 3y ago
That's not the (only) problem, it's that no one can describe what AGI will really look like outside of vague and circular definitions The basic description is "a computer system that can do all relevant things a human being can do". And sure, we don't have a very codified definition of what those things are but we, human beings, exist and so even if we can't define our own capacities, we have an strong idea they exist. The implicit argument is that one can't engineer something one doesn't understand. I tend to agree in the case of full capacities but LLMs seem to have some capacities that their creator don't understand, so the claim doesn't seem a "slam dunk" anymore.
- soco 3y ago
  I feel all these discussion about AGI vs not-AGI are nice and fascinating but useless. MBAs will take whatever EY tells them is smart enough in a fancy powerpoint, kick all human employees out to increase shareholders value and let that expensive computer handle all the red buttons. It doesn't take an AGI to f'up humanity, it only takes a dumb computer trusted with the wrong buttons.
- fl7305 3y ago
  > The implicit argument is that one can't engineer something one doesn't understand. Isn't that easy to debunk? I'd be hard pressed to implement an algorithm that can control a double-jointed inverted pendulum without reading up on control theory again. But I can instead read up on neural networks and implement one that can be trained to control the pendulum. For a more advanced example, I would never be able implement speech recognition explicitly by writing a regular program. But I could probably implement a passable one using neural networks and lots of training material. How is this not "engineer something one doesn't understand"?
  - deleted 3y ago
    [deleted]
- calamari4065 3y ago
  > The implicit argument is that one can't engineer something one doesn't understand. That's what machine learning is. We build these systems we don't fully understand, and set them on solving problems we don't understand. It's revolutionary because it lets us solve problems we don't understand. We don't really fully understand what they do or how, but it does still work.
harpiaharpyja 3y ago
> "That's not the (only) problem, it's that no one can describe what AGI will really look like outside of vague and circular definitions." Not so much unlike the time travel and rayguns, isn't it?

ghostzilla 3y ago

The best description of AGI I've seen is in Michael Crichton's Prey. He describes in a fairly plausible detail how AGI rises from the building blocks. The emotion he captures is that of dealing with a psychopath -- goal oriented with no empathy -- to the infinite degree. That said, I think ChatGPT and LLMs trick us into seeing intelligence where there is none because we are evolutionarily wired to look for signs of intelligence as a potential threat. Our defense mechanisms are asymmetrical as they have to err on the side of being overly cautious: you may mistake a rock for a bear but you won't mistake a bear for a rock. So it is with ChatGPT: its conversation style triggers a false alarm deep in us that there is intelligence there and we can't shake that warning off, but this software design is just exploiting the flaw (of sorts) in our wiring.

candiddevmike 3y ago
> goal oriented with no empathy We'll know we have discovered AGI when it gives us a prompt instead of us providing one.

jdmoreira 3y ago

Sounds like something really simple to define and grasp. What's wrong with, for example: "A software that beats most humans on most cognitive tasks". People make it harder than it needs to he IMO

dragontamer 3y ago
We've had computers that beat humans in differential equations, automated theorems, arithmetic, aiming a gun and more in the 1970s. Even search was considered an AI before Google, but today is just a thing computers are and always have been faster than humans at. Even before ChatGPT, lawyers relied upon computers to double-check citations. Even in programming, I expect that compilers and IDEs check for the most common errors.
- somethingor 3y ago
  I could beat the differential equations AI at aiming a gun so badly
- archargelod 3y ago
  That's why it's Artificial *General* Intelligence. We knew 30 years ago that computer can beat human in the field it was engineered for. But AGI is good at anything you throw at it, including building a better AI. Now give it agency and that's a recipe for a disaster.
- namibj 3y ago
  We still don't beat humans in e.g. LeanDojo, to my understanding. But we also haven't yet thrown the state of the art in graph attention technology at doing the core task of in-context learning (consider the entire explored (partial-)proof "tree" for deciding what tactic with which premises (if the tactic takes an AST/expression argument, like "use that definition/equivalence there as a rewrite rule on the current goal at this node in the proof tree") to apply next at which node in the (partial-)proof tree) in the interactive theorem proving situation, let alone with reinforcement learning that uses the attention structures to funnel back information on what things in dead branches had responsibility for the decisions that made the non-dead branches. Because the RL agent should happily use the theorem prover as a "calculator", and only get "charged" for how much the ML inference and the actual cheap calculator execution really cost. Exploration in-context is what humans also do, and immensely powerful. Similar to, when programming, testing some expressions to figure out edge-cases before incorporating them into a large function/algorithm that has poor observability. So far, machines still mostly do the task of "computing" in theorem proving for us, rather than show some proper intelligence. It's damming that LeanDojo got so high marks without reinforcement learning. They pretty much just dumped prover context state during a run through the entire matlib "standard" library and handed it to two transformers, one for embedding premises and one encoder-decoder fed with the current state and as many premises as the encoder context fits after the state, generating/sampling a few tactics to apply. No freedom of where in the proof tree to apply the next tactic, no awareness of any such side-branches.

joe_the_user 3y ago

That's not the (only) problem, it's that no one can describe what AGI will really look like outside of vague and circular definitions The basic description is "a computer system that can do all relevant things a human being can do". And sure, we don't have a very codified definition of what those things are but we, human beings, exist and so even if we can't define our own capacities, we have an strong idea they exist. The implicit argument is that one can't engineer something one doesn't understand. I tend to agree in the case of full capacities but LLMs seem to have some capacities that their creator don't understand, so the claim doesn't seem a "slam dunk" anymore.

soco 3y ago
I feel all these discussion about AGI vs not-AGI are nice and fascinating but useless. MBAs will take whatever EY tells them is smart enough in a fancy powerpoint, kick all human employees out to increase shareholders value and let that expensive computer handle all the red buttons. It doesn't take an AGI to f'up humanity, it only takes a dumb computer trusted with the wrong buttons.
fl7305 3y ago
> The implicit argument is that one can't engineer something one doesn't understand. Isn't that easy to debunk? I'd be hard pressed to implement an algorithm that can control a double-jointed inverted pendulum without reading up on control theory again. But I can instead read up on neural networks and implement one that can be trained to control the pendulum. For a more advanced example, I would never be able implement speech recognition explicitly by writing a regular program. But I could probably implement a passable one using neural networks and lots of training material. How is this not "engineer something one doesn't understand"?
- deleted 3y ago
  [deleted]
calamari4065 3y ago
> The implicit argument is that one can't engineer something one doesn't understand. That's what machine learning is. We build these systems we don't fully understand, and set them on solving problems we don't understand. It's revolutionary because it lets us solve problems we don't understand. We don't really fully understand what they do or how, but it does still work.

harpiaharpyja 3y ago

> "That's not the (only) problem, it's that no one can describe what AGI will really look like outside of vague and circular definitions." Not so much unlike the time travel and rayguns, isn't it?