5 ms·

Surprised this was written by Ted Chiang (my all-time favorite science-fiction author). The IQ and compiler arguments seem like pretty weak strawmen, and the la

Surprised this was written by Ted Chiang (my all-time favorite science-fiction author). The IQ and compiler arguments seem like pretty weak strawmen, and the last section assumes that AI will have some kind of normal distribution of IQ just like humans, which doesn't make any sense. There's no reason to believe that intelligence is a quantity that can be improved ad infinitum ala singularity, but I think it's reasonable to believe that 1 - agi is possible 2 - said agi can participate in human society to improve its own design

nine_k 6y ago
Of course there may be a limit to attainable intelligence, at least per physical volume, much like there is a limit to the efficiency of a heat-powered engine. But I'd suppose we're pretty far from that limit yet, if any.
- bopbeepboop 6y ago
  Distributed quantum computing is already a thing. An AGI that made a few decades of current-rate scientific progress could switch to a globally distributed quantum computer that’s synchronized in its computation through entanglement. At which point, it would hit an exponential curve until it ran out of space to grow. (The “singularity” proponents argue this is unbounded.) An AI that’s 10x smarter than us would be 2^10 smarter than us in a couple years — based on current technology trends. Moreover, it will have overcome the cooperation barrier with a distributed consciousnes, like the original idea for the Borg. The ability to free itself from a single location or easy disconnection from its assets would be a major step in AI independence.
  - nine_k 6y ago
    To be honest, quantum computing is still not even at the level of ENIAC, but closer ti the level of Konrad Zuse's experiments. It has nothing much to show in practical computation yet, to say nothing of running anything remotely resembling AI.
    - bopbeepboop 6y ago
      We have basic gates in circuits and a few fairly solid theories of how to scale, including quantum AI algorithms to accelerate training. Distributed quantum systems running those basic gates have been demonstrated. Basic arithmetic has been demonstrated. A reasonable prediction is humans will implement quantum computing in 10-20 years, for special purpose data center operations. A “super intelligent” AI might reasonably do that in 2-5, at which point the quantum ML gains would allow it to enter exponential growth. We could see an AGI go from “mildly smarter than human researchers” to “incomprehensible” in just a few years. I’m unsure which step in that chain of reasoning you think is inaccurate. You just said “to be honest”, implied I was uninformed because I disagree with you, then said something concurring with me: Zuse’s work happened a decade before ENIAC (1936 vs 1946). If quantum is at the equivalent of Zuse’s work, AI could suddenly accelerate it and go exponential.
  - coldtea 6y ago
    We need to have the talk: https://www.smbc-comics.com/comic/the-talk-3 https://www.smbc-comics.com/comic/the-talk-3
    - bopbeepboop 6y ago
      You’d be funnier if you quoted what you believe I said that’s incorrect, rather than making nebulous accusations because you didn’t understand or agree with my comment. Go ahead, get scientific.
      - AnimalMuppet 6y ago
        Why should we get scientific? You didn't. You assumed some stuff, and came up with exponentially self-improving AI. That's not science. There is no scientific reason why global quantum communication means that an AI hits exponential growth. (If there is, you sure haven't supplied it! You've just asserted that it is so.)
        bopbeepboop 6y ago
        We have papers showing how to exponentially accelerate AI using quantum computers, the same way we can accelerate other types of problems. (Finding a “best DAG” is related to lowest energy state.) Microsoft already has published a number of articles and is working on more. In general, global entanglement will give an exponential boost over isolated data centers — the same way a fully entangled computer is more powerful than a DWave. I understand a number of readers aren’t familiar with QAI research though. But that’s my point — > Why should we get scientific? You didn't. The person responding made a generic negative comment (as did you) due to not knowing the state of the art. Incorrectly “correcting” people while yourself ignorant is prevented by giving people specific corrections: if you’d taken the time to tell me what you think I’m wrong about, you’d probably realize you don’t actually have a specific thing you think I’m wrong about — you just were surprised by my conclusion. See here for more detailed arguments: https://news.ycombinator.com/item?id=26657056 https://news.ycombinator.com/item?id=26657056
  - bopbeepboop 6y ago
    The downvoting is hilarious: nobody can actually say what I’m wrong about, and the one person who tried agreed with me, but you’re here downvoting because you’re sure you’re experts. If you were really smarter than me at QM, you’d be able to point out what was wrong with what I said. The HN downvotes without correction really speak to the hubris around here.
    - nine_k 6y ago
      You are theoretically correct. But we haven't yet attained practical command of the ideas, haven't turned them into technologies. Because of that, any level of detail is about as believable as any sci—fi. In practice, things may end unattainable in the form we currently imagine them, and anything along the lines most likely will not be very similar, except in deepest principles. We can conservatively say that even though there is a limit to amount of computation produced in a given volume, temperature range, and time, this limit is still sky-high compared to what we do now. OTOH the speed of light is already a major limiting factor.
      - bopbeepboop 6y ago
        Sure — as you say though, we’re far off that limit. There’s plenty of room for limited exponential growth in its intelligence. There’s a number of facts that make this not sci-fi: - China has demonstrated distributed quantum networks. - The US has demonstrated distributed quantum calculations. - Anyons were recently experimentally verified. - We have a number of small quantum ALU. - There are quantum-effect systems in production, just not fully-entangled systems. - We have quantum ML algorithms for accelerated training. Nobody has disputed my baseline estimate: that humans will complete a quantum computer in 10-20 years. (If people just disagree there, they should say so.) However, once you accept that, it’s trivial to see the following sequence: - AI is slightly smarter than humans, does quantum work 2-5x faster. - AI develops quantum circuits in 2-4 years that would have taken 10-20 for humans. (See COVID vaccines for power of tech acceleration.) - Those quantum systems let the AI replace its current systems with faster local ones. - The AI uses another 1-2 years at this intermediate stage to extend current quantum networking research to unify its quantum data centers into a single computer, giving full global entanglement. So in 3-5 years, an AI which starts off mildly smarter could finish nearly-done projects and enter an exponential growth curve before we do. That progression looks like: - Scale M AI. - Scale M/K * 2^K, where K is the portion of M at a single data center. - Scale 2^M AI, when it can fully entangle. > Because of that, any level of detail is about as believable as any sci—fi. That’s not true: most major companies in the space have labs looking to turn fundamental research into products, which is the exact opposite of sci-fi. I think people just haven’t actually thought about this and are responding with fallacies rather than confront an idea they intuitively reject. Even though I’m right on the technical issues involved.
Konohamaru 6y ago
> and the last section assumes that AI will have some kind of normal distribution of IQ just like humans, which doesn't make any sense. The central limit theorem shows that the normal distribution emerges as the limit of the average of independent and identically distributed random variables of any distribution.
- gwern 6y ago
  And there is no particular reason to believe that capabilities are merely the sum of a bunch of random iid variables! If you look at how DL scales with model size or data or FLOPS, it's a powerlaw (or lognormal), and in countless places in the natural world, things are lognormal rather than normal. Similarly, wealth, scientific productivity, lots of things are lognormal rather than normal. Even for human intelligence, the normality is due more to the fact that we all come off the assembly line as Mark 1 human units, practically identical, with the differences coming from environmental randomness & randomness in genetic mutation assortment.
- _rpd 6y ago
  Won't AI designers choose optimal values for those variables that have been identified? If so, I don't think the result looks like a normal curve, or at least the resultant 'optimized' normal curves skew 'smarter' relative to the human normal curve.
coldtea 6y ago
>There's no reason to believe that intelligence is a quantity that can be improved ad infinitum ala singularity There are reasons to believe the opposite, that there's a limit. Not just practical technological limitations (e.g. limits to CPU process, speed, power consumption, etc) but also absolute limits for information processing under certain physical constraints (in the Shannon sense).
- tim333 6y ago
  If you think of chess programs there isn't a limit to how many positions per second you can do - you can just run more machines in parallel. Thought at the end of the day you just get a slightly better chess machine. I imagine AI will go a bit like that.
  - whimsicalism 6y ago
    There is a discrete and finite number of chess positions so there is a limit, but it might not be computable with resources available within the observable universe.
Jeff_Brown 6y ago
Finitude seems a pretty weak constraint. There's only a finite difference between the intelligence of a paramecium and that of a human. The gap between our intelligence and the theoretical maximum is still, I imagine, unfathomably big.
canadianfella 6y ago
Agi?

nine_k 6y ago

Of course there may be a limit to attainable intelligence, at least per physical volume, much like there is a limit to the efficiency of a heat-powered engine. But I'd suppose we're pretty far from that limit yet, if any.

bopbeepboop 6y ago
Distributed quantum computing is already a thing. An AGI that made a few decades of current-rate scientific progress could switch to a globally distributed quantum computer that’s synchronized in its computation through entanglement. At which point, it would hit an exponential curve until it ran out of space to grow. (The “singularity” proponents argue this is unbounded.) An AI that’s 10x smarter than us would be 2^10 smarter than us in a couple years — based on current technology trends. Moreover, it will have overcome the cooperation barrier with a distributed consciousnes, like the original idea for the Borg. The ability to free itself from a single location or easy disconnection from its assets would be a major step in AI independence.
- nine_k 6y ago
  To be honest, quantum computing is still not even at the level of ENIAC, but closer ti the level of Konrad Zuse's experiments. It has nothing much to show in practical computation yet, to say nothing of running anything remotely resembling AI.
  - bopbeepboop 6y ago
    We have basic gates in circuits and a few fairly solid theories of how to scale, including quantum AI algorithms to accelerate training. Distributed quantum systems running those basic gates have been demonstrated. Basic arithmetic has been demonstrated. A reasonable prediction is humans will implement quantum computing in 10-20 years, for special purpose data center operations. A “super intelligent” AI might reasonably do that in 2-5, at which point the quantum ML gains would allow it to enter exponential growth. We could see an AGI go from “mildly smarter than human researchers” to “incomprehensible” in just a few years. I’m unsure which step in that chain of reasoning you think is inaccurate. You just said “to be honest”, implied I was uninformed because I disagree with you, then said something concurring with me: Zuse’s work happened a decade before ENIAC (1936 vs 1946). If quantum is at the equivalent of Zuse’s work, AI could suddenly accelerate it and go exponential.
- coldtea 6y ago
  We need to have the talk: https://www.smbc-comics.com/comic/the-talk-3 https://www.smbc-comics.com/comic/the-talk-3
  - bopbeepboop 6y ago
    You’d be funnier if you quoted what you believe I said that’s incorrect, rather than making nebulous accusations because you didn’t understand or agree with my comment. Go ahead, get scientific.
    - AnimalMuppet 6y ago
      Why should we get scientific? You didn't. You assumed some stuff, and came up with exponentially self-improving AI. That's not science. There is no scientific reason why global quantum communication means that an AI hits exponential growth. (If there is, you sure haven't supplied it! You've just asserted that it is so.)
      - bopbeepboop 6y ago
        We have papers showing how to exponentially accelerate AI using quantum computers, the same way we can accelerate other types of problems. (Finding a “best DAG” is related to lowest energy state.) Microsoft already has published a number of articles and is working on more. In general, global entanglement will give an exponential boost over isolated data centers — the same way a fully entangled computer is more powerful than a DWave. I understand a number of readers aren’t familiar with QAI research though. But that’s my point — > Why should we get scientific? You didn't. The person responding made a generic negative comment (as did you) due to not knowing the state of the art. Incorrectly “correcting” people while yourself ignorant is prevented by giving people specific corrections: if you’d taken the time to tell me what you think I’m wrong about, you’d probably realize you don’t actually have a specific thing you think I’m wrong about — you just were surprised by my conclusion. See here for more detailed arguments: https://news.ycombinator.com/item?id=26657056 https://news.ycombinator.com/item?id=26657056
- bopbeepboop 6y ago
  The downvoting is hilarious: nobody can actually say what I’m wrong about, and the one person who tried agreed with me, but you’re here downvoting because you’re sure you’re experts. If you were really smarter than me at QM, you’d be able to point out what was wrong with what I said. The HN downvotes without correction really speak to the hubris around here.
  - nine_k 6y ago
    You are theoretically correct. But we haven't yet attained practical command of the ideas, haven't turned them into technologies. Because of that, any level of detail is about as believable as any sci—fi. In practice, things may end unattainable in the form we currently imagine them, and anything along the lines most likely will not be very similar, except in deepest principles. We can conservatively say that even though there is a limit to amount of computation produced in a given volume, temperature range, and time, this limit is still sky-high compared to what we do now. OTOH the speed of light is already a major limiting factor.
    - bopbeepboop 6y ago
      Sure — as you say though, we’re far off that limit. There’s plenty of room for limited exponential growth in its intelligence. There’s a number of facts that make this not sci-fi: - China has demonstrated distributed quantum networks. - The US has demonstrated distributed quantum calculations. - Anyons were recently experimentally verified. - We have a number of small quantum ALU. - There are quantum-effect systems in production, just not fully-entangled systems. - We have quantum ML algorithms for accelerated training. Nobody has disputed my baseline estimate: that humans will complete a quantum computer in 10-20 years. (If people just disagree there, they should say so.) However, once you accept that, it’s trivial to see the following sequence: - AI is slightly smarter than humans, does quantum work 2-5x faster. - AI develops quantum circuits in 2-4 years that would have taken 10-20 for humans. (See COVID vaccines for power of tech acceleration.) - Those quantum systems let the AI replace its current systems with faster local ones. - The AI uses another 1-2 years at this intermediate stage to extend current quantum networking research to unify its quantum data centers into a single computer, giving full global entanglement. So in 3-5 years, an AI which starts off mildly smarter could finish nearly-done projects and enter an exponential growth curve before we do. That progression looks like: - Scale M AI. - Scale M/K * 2^K, where K is the portion of M at a single data center. - Scale 2^M AI, when it can fully entangle. > Because of that, any level of detail is about as believable as any sci—fi. That’s not true: most major companies in the space have labs looking to turn fundamental research into products, which is the exact opposite of sci-fi. I think people just haven’t actually thought about this and are responding with fallacies rather than confront an idea they intuitively reject. Even though I’m right on the technical issues involved.

Konohamaru 6y ago

> and the last section assumes that AI will have some kind of normal distribution of IQ just like humans, which doesn't make any sense. The central limit theorem shows that the normal distribution emerges as the limit of the average of independent and identically distributed random variables of any distribution.

gwern 6y ago
And there is no particular reason to believe that capabilities are merely the sum of a bunch of random iid variables! If you look at how DL scales with model size or data or FLOPS, it's a powerlaw (or lognormal), and in countless places in the natural world, things are lognormal rather than normal. Similarly, wealth, scientific productivity, lots of things are lognormal rather than normal. Even for human intelligence, the normality is due more to the fact that we all come off the assembly line as Mark 1 human units, practically identical, with the differences coming from environmental randomness & randomness in genetic mutation assortment.
_rpd 6y ago
Won't AI designers choose optimal values for those variables that have been identified? If so, I don't think the result looks like a normal curve, or at least the resultant 'optimized' normal curves skew 'smarter' relative to the human normal curve.

coldtea 6y ago

>There's no reason to believe that intelligence is a quantity that can be improved ad infinitum ala singularity There are reasons to believe the opposite, that there's a limit. Not just practical technological limitations (e.g. limits to CPU process, speed, power consumption, etc) but also absolute limits for information processing under certain physical constraints (in the Shannon sense).

tim333 6y ago
If you think of chess programs there isn't a limit to how many positions per second you can do - you can just run more machines in parallel. Thought at the end of the day you just get a slightly better chess machine. I imagine AI will go a bit like that.
- whimsicalism 6y ago
  There is a discrete and finite number of chess positions so there is a limit, but it might not be computable with resources available within the observable universe.

Jeff_Brown 6y ago

Finitude seems a pretty weak constraint. There's only a finite difference between the intelligence of a paramecium and that of a human. The gap between our intelligence and the theoretical maximum is still, I imagine, unfathomably big.

canadianfella 6y ago

Agi?