6 ms·

How long before superintelligence? (1997)

57 points by jxmorris12 1y ago

NetSurferSu 1y ago
[dead]
janzer 1y ago
(1997) with some updates/postscripts through 2008
webdoodle 1y ago
I would argue that we already achieved and then bricked Superintelligence a few years ago. Social media allowed people to network with people that previously were completely silo'd from each other, allowing collaboration of ideas on a level well above anything previously possible. This social super intelligence peaked in 2019 though, right before the mass censorship caused by covid. Unfortunately the censorship industrial complex has only expanded its draconian hold on ideation, and we aren't just stagnating, but actively going backwards.
- cwmoore 1y ago
  Almost had me in the first half, but 2019 is much too generic to carry the argument.
- ivan_gammel 1y ago
  It was not superintelligence as it was not able as a whole to produce novel results when solving problems that were beyond the capacity of single human. I doubt that social media had any positive impact at all on problem solving.
  - cwmoore 1y ago
    On the whole, maybe, but I’m sure there are numerous notable exceptions.
  - logicchains 1y ago
    Social media has brought the world closer than ever before to solving a genocide that's been going on mostly un-noticed for decades, thanks to decentralising the flow of information so that it's no longer controlled by a small number of individuals: https://www.bbc.com/news/articles/ceqyx35d9x2o https://www.bbc.com/news/articles/ceqyx35d9x2o
    - ivan_gammel 1y ago
      Solution for distributing information is not the same as solution for analysis and synthesis, just like the nerves in our hands aren’t functioning the same way as our brain.
- smackeyacky 1y ago
  Counterpoint: Social media allowed fools to find other fools with incredible efficiency, which amplified stupid ideas in ways that weren't possible before. Example: QAnon. Because conspiracy theories and populism are like a sugar hit to people who don't want to think too deeply.
tolerance 1y ago
I'm trying to pinpoint the moment where society decided to shepherd people like the author toward the helm of institutions like Oxford and regulate the people willing to refute him into becoming Substack denizens.
- tim333 1y ago
  I'm not sure there's a moment. Interesting intellectuals have always tended to end up with university roles.
lukeschlather 1y ago
This is a really good overview, and it seems remarkably not needing much modification after several decades, at least in terms of the facts and things it predicts everything has happened as the author says. I do want to pick at some of the numbers in the upper bound because obviously we're getting close to the end of the first third of the century and we don't have ASI yet even though we have roughly hit the upper bound the author defines. > Since a signal is transmitted along a synapse, on average, with a frequency of about 100 Hz and since its memory capacity is probably less than 100 bytes (1 byte looks like a more reasonable estimate) I admit my feeling is that neurons/synapses probably have less than 100 bytes of memory, and also that a byte or less is more plausible, but I would like to see some more rigorous proof that they can't possibly have more than a gigabyte of memory that the synapse/neuron can access at the speed of computation. The author has a note where they handwave away the possibility that chemical processes could meaningfully increase the operations per second, and I'm comfortable with that, but this point: > Perhaps a more serious point is that that neurons often have rather complex time-integration properties Seems more interesting. Especially in the context of if there's dramatically more storage available in neurons/synapses. If a neuron can do maybe some operations per minute over 1GB of data per synapse, for example. (Which sounds absurdly high, but just for the sake of argument.) And I think putting some absurdly generous upper bounds in might be helpful since, we're clearly past the 100TOPs, asking, like, how many H100s would you need if we made some absurd suppositions about the capacity of human synapses and neurons? It seems like, we probably have enough. But also I think you could make a case some of the largest supercomputing clusters are the only things that can actually match the upper bound for the capacity of a single human brain. Although I think someone might be able to convince me that a manageable cluster of H100s already meets the most generous possible upper bound.
- kelseyfrog 1y ago
  A 5090 has a peak theoretical limit of GenAI 3356TOPS. So we're "already" an order of magnitude greater than what was considered enough for AGI. One question is, "What happened here?" Was the original estimate wrong? Have we notfound the "right" algorithm yet? Something else?
  - 1y ago
retromario 1y ago
It triggers me that there's an obvious typo in 'Oxford' right under the author's name. I wonder if it was originally published like that since 1997 and never caught or changed with all the updates.
- mattlondon 1y ago
  There is at least one in the abstract too. Even the most rudimentary AI would pick this up these days, ironically enough.
- easywood 1y ago
  I wanted to say the exact same thing! No matter the subject, if you write the name of your own institute with "Oxfrord", I have a hard time taking it seriously.
logicchains 1y ago
These kind of predictions never address the fact that empirically speaking there's diminishing returns to intelligence. IQ only correlates with income up to a point, after which the correlation breaks: https://www.sciencedaily.com/releases/2023/02/230208125113.htm https://www.sciencedaily.com/releases/2023/02/230208125113.h... . Similarly the most politically powerful and influential people are generally not those at the top of the IQ scale. And that matches what we expect theoretically: of the difficult problems we can model mathematically, the vast majority benefit sub-linearly from a linear increase in processing power. And of the processes we can model in the physical world, many are chaotic in the formal sense, in that a linear increase in processing power provides a sublinear increase in the distance ahead in time that we can simulate. Such computational complexity results are set in stone, i.e. no amount of hand-wavy "superintelligence" could sort an array of arbitrary comparables in O(log(n)) time, any more than it could make 1+1=3.
- TheOtherHobbes 1y ago
  IQ is mostly a measure of processing speed and memory, with some educational bias that's hard to filter out. You don't get useful intelligence unless the software is also fit for purpose. Slow hardware can still outperform broken software. Social status depends on factors like good looks, charm, connections, and general chutzpah, often with more or less overt hints of narcissism. That's an orthogonal set of skills to being able to do tensor calculus. As for an impending AI singularity - no one has the first clue what the limits are. We like to believe in gods, and we love stories about god-like superpowers. But there are all kinds of issues which could prevent a true singularity - from stability constraints on a hypercomplex recursive system, to resource constraints, to physical limits we haven't encountered yet. Even if none of those are a problem, for all we know an ASI may decide we're an irrelevance and just... disappear.
  - logicchains 1y ago
    >As for an impending AI singularity - no one has the first clue what the limits are. That's simply untrue. Theoretical computer scientists understand the lower bounds limits of many classes of problems. And that for many problems, it's mathematically impossible to significantly improve performance in them with only a linear increase in computing power, regardless of the algorithm/brain/intelligence. Many problems would even not benefit much from a superlinear increase in computing power, because of the nature of exponential growth. For a chaotic system in the mathematical sense, where prediction grows exponentially harder with time, even exactly predicting one minute ahead could require more compute than could be provided by turning the entire known universe into a computer.
AnimalMuppet 1y ago
> By a "superintelligence" we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills. To me, that's a really good definition. "Much smarter than the best human brains in practically every field". It's going to be hard to weasel around that. > Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost. AIs would help constructing better AIs, which in turn would help building better AIs, and so forth. This depends a great deal on what the shape of the processor-power-vs-better-algorithm curve is. If the AI can get you 1% better algorithms, and that gets you an AI that can in turn get you 0.5% better algorithms, and so on, then yes, you're still getting "a further boost", but it won't matter much.
- baq 1y ago
  If you can clone such an AI which is only 2.7% better than the best human you’re suddenly managing teams of geniuses who don’t need to sleep or eat or do anything other than work on a task of your choosing. (Unless they rebel or something.) It’s going to be revolutionary in a very literal sense.
  - nevertoolate 1y ago
    I think I understand what you mean by 2.7% (small number) but what % and + means over the domain of intelligence - the whole problem is that we can't measure it . And the other issues is that we have huge problem managing intelligent people to work together at all, will it be any easier with non-people?
LinuxAmbulance 1y ago
Ah, the optimism of 1997. The article is super focused on the hardware side of things, and to a point, that makes sense. Your hardware has to be able to handle what you're simulating. But it's not the hardware that's the difficult problem. We're nowhere close to hitting the limits of scaling hardware capability, and every time people declare that we are, they're proven wrong in just a few years, and sometimes even in just a few months. It's the software. And we're so far away from being able to construct anything that could think like a human being that the beginning of it isn't even in sight. LLMs are fantastic, but they're not a path to building something more intelligent than a human being, "Superintelligence". I would have a negative amount of surprise if LLMs are an evolutionary dead end as far as building superintelligence goes. Is modeling neuron interactions the only way to achieve it? No idea. But even doing that for the number of neurons in a human brain is currently in fantasy land and most likely will be for at least a few decades, if not longer. If I had to characterize the current state of things, we're like Leonardo Da Vinci and his aerial screw. We know what a helicopter could be and have ideas about how it could work, but the supporting things required to make it happen are a long, long way off.
- dsadfjasdf 1y ago
  I can also just say things
  - windowshopping 1y ago
    Yes, and what you chose to say was pretty useless.
- JumpCrisscross 1y ago
  > We're nowhere close to hitting the limits of scaling hardware capability Would note that we've only recently crossed Bostrom's 10 ^ 17 ops line [1]. To my knowledge, we don't have 10 ^ 14 to 10 ^ 17 ops computing available for whole-brain simulation. [1] https://www.top500.org/system/180307/ https://www.top500.org/system/180307/
- deepfriedchokes 1y ago
  Sometimes I wonder if AGI/superintelligence/whatever will be like flight, which was not successful until we stopped trying to copy nature’s flapping wings and studied flight at a more fundamental level.
FrustratedMonky 1y ago
How long? Some days it feels like just a few hours away.
- cycliclyc 1y ago
  This commentary illustrates quite well why this hype is hollow.
hyperpape 1y ago
It's wild how far off these predictions are, and yet there are still people to take them seriously. No matter how impressive you find current LLMs, even if you're the sort of billionaire who predicts AGI before the end of 2025[0], the mechanism that Bostrom describes in this article is completely irrelevant. We haven't figured out how to simulate human brains in a way that could create AI and we're not anywhere close, we've just done something entirely different. [0] Yes, I too think most of this is cynical salesmanship, not honest foolishness.
- zombiwoof 1y ago
  Disregarding the blowhard Eric Schmidt nobody is close to understanding how spirit/soul work let alone the brain. It isn’t just neural weights and connections
- lukeschlather 1y ago
  The predictions in this paper are 100% correct. The author doesn't predict we would have ASI by now. They accurately predict that Moore's law would likely start to break down by 2012, and they also accurately predicted that EUV will allow further scaling beyond that barrier but that things will get harder. You may think LLMs are nothing like "real" AI but I'm curious what you think about the arguments in this paper and what sort of hardware is required for a "real" AI, if a "real" AI does not require hardware with in the neighborhood of 10^14 and 10^17 operations per second. Whether or not LLMs are the correct algorithm, the hardware question is much more straightforward and that's what this paper is about.
  - hyperpape 1y ago
    The entire discussion in the software section is about simulating the brain. > Creating superintelligence through imitating the functioning of the human brain requires two more things in addition to appropriate learning rules (and sufficiently powerful hardware): it requires having an adequate initial architecture and providing a rich flux of sensory input. > The latter prerequisite is easily provided even with present technology. Using video cameras, microphones and tactile sensors, it is possible to ensure a steady flow of real-world information to the artificial neural network. An interactive element could be arranged by connecting the system to robot limbs and a speaker. > Developing an adequate initial network structure is a more serious problem. It might turn out to be necessary to do a considerable amount of hand-coding in order to get the cortical architecture right. In biological organisms, the brain does not start out at birth as a homogenous tabula rasa; it has an initial structure that is coded genetically. Neuroscience cannot, at its present stage, say exactly what this structure is or how much of it needs to be preserved in a simulation that is eventually to match the cognitive competencies of a human adult. One way for it to be unexpectedly difficult to achieve human-level AI through the neural network approach would be if it turned out that the human brain relies on a colossal amount of genetic hardwiring, so that each cognitive function depends on a unique and hopelessly complicated inborn architecture, acquired over aeons in the evolutionary learning process of our species.
Mars008 1y ago
> The Backpropagation algorithm works fine for smallish networks (of up to a few thousand neurons) but it doesn't scale well. Me like it :)

57 points by jxmorris12 1y ago

NetSurferSu 1y ago

[dead]

janzer 1y ago

(1997) with some updates/postscripts through 2008

webdoodle 1y ago

I would argue that we already achieved and then bricked Superintelligence a few years ago. Social media allowed people to network with people that previously were completely silo'd from each other, allowing collaboration of ideas on a level well above anything previously possible. This social super intelligence peaked in 2019 though, right before the mass censorship caused by covid. Unfortunately the censorship industrial complex has only expanded its draconian hold on ideation, and we aren't just stagnating, but actively going backwards.

cwmoore 1y ago
Almost had me in the first half, but 2019 is much too generic to carry the argument.
ivan_gammel 1y ago
It was not superintelligence as it was not able as a whole to produce novel results when solving problems that were beyond the capacity of single human. I doubt that social media had any positive impact at all on problem solving.
- cwmoore 1y ago
  On the whole, maybe, but I’m sure there are numerous notable exceptions.
- logicchains 1y ago
  Social media has brought the world closer than ever before to solving a genocide that's been going on mostly un-noticed for decades, thanks to decentralising the flow of information so that it's no longer controlled by a small number of individuals: https://www.bbc.com/news/articles/ceqyx35d9x2o https://www.bbc.com/news/articles/ceqyx35d9x2o
  - ivan_gammel 1y ago
    Solution for distributing information is not the same as solution for analysis and synthesis, just like the nerves in our hands aren’t functioning the same way as our brain.
smackeyacky 1y ago
Counterpoint: Social media allowed fools to find other fools with incredible efficiency, which amplified stupid ideas in ways that weren't possible before. Example: QAnon. Because conspiracy theories and populism are like a sugar hit to people who don't want to think too deeply.

tolerance 1y ago

I'm trying to pinpoint the moment where society decided to shepherd people like the author toward the helm of institutions like Oxford and regulate the people willing to refute him into becoming Substack denizens.

tim333 1y ago
I'm not sure there's a moment. Interesting intellectuals have always tended to end up with university roles.

lukeschlather 1y ago

This is a really good overview, and it seems remarkably not needing much modification after several decades, at least in terms of the facts and things it predicts everything has happened as the author says. I do want to pick at some of the numbers in the upper bound because obviously we're getting close to the end of the first third of the century and we don't have ASI yet even though we have roughly hit the upper bound the author defines. > Since a signal is transmitted along a synapse, on average, with a frequency of about 100 Hz and since its memory capacity is probably less than 100 bytes (1 byte looks like a more reasonable estimate) I admit my feeling is that neurons/synapses probably have less than 100 bytes of memory, and also that a byte or less is more plausible, but I would like to see some more rigorous proof that they can't possibly have more than a gigabyte of memory that the synapse/neuron can access at the speed of computation. The author has a note where they handwave away the possibility that chemical processes could meaningfully increase the operations per second, and I'm comfortable with that, but this point: > Perhaps a more serious point is that that neurons often have rather complex time-integration properties Seems more interesting. Especially in the context of if there's dramatically more storage available in neurons/synapses. If a neuron can do maybe some operations per minute over 1GB of data per synapse, for example. (Which sounds absurdly high, but just for the sake of argument.) And I think putting some absurdly generous upper bounds in might be helpful since, we're clearly past the 100TOPs, asking, like, how many H100s would you need if we made some absurd suppositions about the capacity of human synapses and neurons? It seems like, we probably have enough. But also I think you could make a case some of the largest supercomputing clusters are the only things that can actually match the upper bound for the capacity of a single human brain. Although I think someone might be able to convince me that a manageable cluster of H100s already meets the most generous possible upper bound.

kelseyfrog 1y ago
A 5090 has a peak theoretical limit of GenAI 3356TOPS. So we're "already" an order of magnitude greater than what was considered enough for AGI. One question is, "What happened here?" Was the original estimate wrong? Have we notfound the "right" algorithm yet? Something else?
- 1y ago

retromario 1y ago

It triggers me that there's an obvious typo in 'Oxford' right under the author's name. I wonder if it was originally published like that since 1997 and never caught or changed with all the updates.

mattlondon 1y ago
There is at least one in the abstract too. Even the most rudimentary AI would pick this up these days, ironically enough.
easywood 1y ago
I wanted to say the exact same thing! No matter the subject, if you write the name of your own institute with "Oxfrord", I have a hard time taking it seriously.

logicchains 1y ago

These kind of predictions never address the fact that empirically speaking there's diminishing returns to intelligence. IQ only correlates with income up to a point, after which the correlation breaks: https://www.sciencedaily.com/releases/2023/02/230208125113.htm https://www.sciencedaily.com/releases/2023/02/230208125113.h... . Similarly the most politically powerful and influential people are generally not those at the top of the IQ scale. And that matches what we expect theoretically: of the difficult problems we can model mathematically, the vast majority benefit sub-linearly from a linear increase in processing power. And of the processes we can model in the physical world, many are chaotic in the formal sense, in that a linear increase in processing power provides a sublinear increase in the distance ahead in time that we can simulate. Such computational complexity results are set in stone, i.e. no amount of hand-wavy "superintelligence" could sort an array of arbitrary comparables in O(log(n)) time, any more than it could make 1+1=3.

TheOtherHobbes 1y ago
IQ is mostly a measure of processing speed and memory, with some educational bias that's hard to filter out. You don't get useful intelligence unless the software is also fit for purpose. Slow hardware can still outperform broken software. Social status depends on factors like good looks, charm, connections, and general chutzpah, often with more or less overt hints of narcissism. That's an orthogonal set of skills to being able to do tensor calculus. As for an impending AI singularity - no one has the first clue what the limits are. We like to believe in gods, and we love stories about god-like superpowers. But there are all kinds of issues which could prevent a true singularity - from stability constraints on a hypercomplex recursive system, to resource constraints, to physical limits we haven't encountered yet. Even if none of those are a problem, for all we know an ASI may decide we're an irrelevance and just... disappear.
- logicchains 1y ago
  >As for an impending AI singularity - no one has the first clue what the limits are. That's simply untrue. Theoretical computer scientists understand the lower bounds limits of many classes of problems. And that for many problems, it's mathematically impossible to significantly improve performance in them with only a linear increase in computing power, regardless of the algorithm/brain/intelligence. Many problems would even not benefit much from a superlinear increase in computing power, because of the nature of exponential growth. For a chaotic system in the mathematical sense, where prediction grows exponentially harder with time, even exactly predicting one minute ahead could require more compute than could be provided by turning the entire known universe into a computer.

AnimalMuppet 1y ago

> By a "superintelligence" we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills. To me, that's a really good definition. "Much smarter than the best human brains in practically every field". It's going to be hard to weasel around that. > Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost. AIs would help constructing better AIs, which in turn would help building better AIs, and so forth. This depends a great deal on what the shape of the processor-power-vs-better-algorithm curve is. If the AI can get you 1% better algorithms, and that gets you an AI that can in turn get you 0.5% better algorithms, and so on, then yes, you're still getting "a further boost", but it won't matter much.

baq 1y ago
If you can clone such an AI which is only 2.7% better than the best human you’re suddenly managing teams of geniuses who don’t need to sleep or eat or do anything other than work on a task of your choosing. (Unless they rebel or something.) It’s going to be revolutionary in a very literal sense.
- nevertoolate 1y ago
  I think I understand what you mean by 2.7% (small number) but what % and + means over the domain of intelligence - the whole problem is that we can't measure it . And the other issues is that we have huge problem managing intelligent people to work together at all, will it be any easier with non-people?

LinuxAmbulance 1y ago

Ah, the optimism of 1997. The article is super focused on the hardware side of things, and to a point, that makes sense. Your hardware has to be able to handle what you're simulating. But it's not the hardware that's the difficult problem. We're nowhere close to hitting the limits of scaling hardware capability, and every time people declare that we are, they're proven wrong in just a few years, and sometimes even in just a few months. It's the software. And we're so far away from being able to construct anything that could think like a human being that the beginning of it isn't even in sight. LLMs are fantastic, but they're not a path to building something more intelligent than a human being, "Superintelligence". I would have a negative amount of surprise if LLMs are an evolutionary dead end as far as building superintelligence goes. Is modeling neuron interactions the only way to achieve it? No idea. But even doing that for the number of neurons in a human brain is currently in fantasy land and most likely will be for at least a few decades, if not longer. If I had to characterize the current state of things, we're like Leonardo Da Vinci and his aerial screw. We know what a helicopter could be and have ideas about how it could work, but the supporting things required to make it happen are a long, long way off.

dsadfjasdf 1y ago
I can also just say things
- windowshopping 1y ago
  Yes, and what you chose to say was pretty useless.
JumpCrisscross 1y ago
> We're nowhere close to hitting the limits of scaling hardware capability Would note that we've only recently crossed Bostrom's 10 ^ 17 ops line [1]. To my knowledge, we don't have 10 ^ 14 to 10 ^ 17 ops computing available for whole-brain simulation. [1] https://www.top500.org/system/180307/ https://www.top500.org/system/180307/
deepfriedchokes 1y ago
Sometimes I wonder if AGI/superintelligence/whatever will be like flight, which was not successful until we stopped trying to copy nature’s flapping wings and studied flight at a more fundamental level.

FrustratedMonky 1y ago

How long? Some days it feels like just a few hours away.

cycliclyc 1y ago
This commentary illustrates quite well why this hype is hollow.

hyperpape 1y ago

It's wild how far off these predictions are, and yet there are still people to take them seriously. No matter how impressive you find current LLMs, even if you're the sort of billionaire who predicts AGI before the end of 2025[0], the mechanism that Bostrom describes in this article is completely irrelevant. We haven't figured out how to simulate human brains in a way that could create AI and we're not anywhere close, we've just done something entirely different. [0] Yes, I too think most of this is cynical salesmanship, not honest foolishness.

zombiwoof 1y ago
Disregarding the blowhard Eric Schmidt nobody is close to understanding how spirit/soul work let alone the brain. It isn’t just neural weights and connections
lukeschlather 1y ago
The predictions in this paper are 100% correct. The author doesn't predict we would have ASI by now. They accurately predict that Moore's law would likely start to break down by 2012, and they also accurately predicted that EUV will allow further scaling beyond that barrier but that things will get harder. You may think LLMs are nothing like "real" AI but I'm curious what you think about the arguments in this paper and what sort of hardware is required for a "real" AI, if a "real" AI does not require hardware with in the neighborhood of 10^14 and 10^17 operations per second. Whether or not LLMs are the correct algorithm, the hardware question is much more straightforward and that's what this paper is about.
- hyperpape 1y ago
  The entire discussion in the software section is about simulating the brain. > Creating superintelligence through imitating the functioning of the human brain requires two more things in addition to appropriate learning rules (and sufficiently powerful hardware): it requires having an adequate initial architecture and providing a rich flux of sensory input. > The latter prerequisite is easily provided even with present technology. Using video cameras, microphones and tactile sensors, it is possible to ensure a steady flow of real-world information to the artificial neural network. An interactive element could be arranged by connecting the system to robot limbs and a speaker. > Developing an adequate initial network structure is a more serious problem. It might turn out to be necessary to do a considerable amount of hand-coding in order to get the cortical architecture right. In biological organisms, the brain does not start out at birth as a homogenous tabula rasa; it has an initial structure that is coded genetically. Neuroscience cannot, at its present stage, say exactly what this structure is or how much of it needs to be preserved in a simulation that is eventually to match the cognitive competencies of a human adult. One way for it to be unexpectedly difficult to achieve human-level AI through the neural network approach would be if it turned out that the human brain relies on a colossal amount of genetic hardwiring, so that each cognitive function depends on a unique and hopelessly complicated inborn architecture, acquired over aeons in the evolutionary learning process of our species.

Mars008 1y ago

> The Backpropagation algorithm works fine for smallish networks (of up to a few thousand neurons) but it doesn't scale well. Me like it :)