4 ms·

> To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes, we can", then there is little

> To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes, we can", then there is little doubt that beings more intelligent can be constructed shortly thereafter. I find it bizarre how often these points are repeated. They were both obviously wrong in 1993, and obviously wrong now. 1) A nitpick I've had since grad school: the answer to "can we create a machine equivalent to a human mind [assuming arbitrary resources]?" is "yes, of course." The atoms in a human body can be described by a hideously ugly system of Schrödinger equations and a Turing machine can solve that to arbitrary numerical precision. Even Penrose's loopy stuff about consciousness doesn't change this. QED. 2) The more serious issue: I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. It is bizarre that this claim is accepted with "little doubt" when there are very good reasons to doubt it: how on earth would such an AI even know it succeeded? How would it define the goal? This idea makes sense for improving Steven Tyler-level AI to Thelonious Monk-level; it makes no sense for a transition like chimp->human. Yet that is precisely the magnitude of transition envisioned with these singularity stories. You might defend the first point by emphasizing "can we create a human-level AI?" i.e. not whether it's theoretically possible, but humanly feasible. This just makes the second point even more incoherent! If humans are too stoopid to build a human-level AI, why would a human-level AI be...smarter than us? I just don't understand how anyone can rationally accept this stuff! It's so dumb! Tech folks (and too many philosophers) are hopped up on science fiction: the reason these things are accepted with "little doubt" is that this is religious faith dressed up in the language of science.

bbor 2y ago
If you're ever stuck wondering why a bunch of smart, motivated people with no clear corrupting motivations are being idiotic, that's a strong heuristic that you should spend a bit more time analyzing the issue, IMO ;). "Ugh, why is everyone else so stupid" is a common take for undergrad engineers, but I'm sure you've grown out of it in other ways. Anyway, more substantively: The simple answer is that people have thought about it in depth, most famously noted doomer Eliezer Yudkowsky in Intelligence Explosion Microeconomics (2013)[1] and its main citation, Irving John Good's Speculations Concerning the First Ultraintelligent Machine (1965)[2]. Another common citation that drops a bit of rigour in the name of approachability is Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies[3]. [ETA: to put it even simpler: a system that improves itself is a (the?) quintessential setup for exponential growth. E.g. compound interest] For the time-bound, the most rigorous treatment of your concern among those three is in Section 3 of Yudkowsky's paper, "From AI to Machine Superintelligence". To list the headings briefly: - Increased computational resources - Communication speed - Increased serial depth (i.e. working memory capacity) - Duplicability (i.e. reliability) - Editability (i.e. we know how computers work) - Goal coordination (this is really just communication speed, again) - Improved rationality (i.e. fewer emotions/accidental instincts getting in the way) Let's drop "human" and "superhuman" for a minute, and just talk about "better computers". I'm assuming you're a software engineer. Don't you see how a real software dev replacement program could be an unimaginable gamechanger for software? Working 24/7, enhancing itself, following TDD perfectly every time, and never ever submitting a PR that isn't rigorously documented and reviewed? All of which only gets better over time, as it develops itself? [1] https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [2] https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e423ee-54e0-4eec-aeca-32b73f851af5/content https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e... [3] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... TL;DR May God have mercy on us all.
- amanaplanacanal 2y ago
  And if somehow we were still able to control such a thing, it's much more likely that the goal would be "make my investors and I as much money as possible" rather than "solve all of humanities problems".
  - bbor 2y ago
    Re: “all of humanities problems”, I think that AI will more productively and safely be deployed in personalized, decentralized contexts. AKA Artificial Self-Instantiation for everyone who wants it, rather than a single Rehoboam god machine[1]. Re: capitalism, this is our chance. The world is about to turn upside down. We must strike while the iron is hot. We don’t need to replace capitalism with communism or any other specific thing; we just need to aggressively question all human hierarchies, and get rid of any that can not justify themselves. A just society will come about piece by piece, in this fashion. This is what Chomsky calls “Anarchy”, which is a much more understandable phrasing than what I was taught in high school, ie “no laws ever of any kind” [1] https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U
- LegionMammal978 2y ago
  To me, the big dubious part is, how do we know that increasing the capacity of a reasoning machine can always increase the quality of its output by an equal factor, even when its task is 'designing the next iteration of a reasoning machine'? It may very well be that the task of 'designing a reasoning machine' has sharply diminishing returns when you throw more resources at it. (E.g., at least so far, any LLM can only have a very incidental impact on designing the next LLM, it's not going to come up with groundbreaking new approaches.) Yudkowsky builds the whole edifice on top of his very particular conception of intelligence, which insists upon itself, but which I think is far from the only explanation of its nature as observed in humans, other animals, LLMs maybe, and so on.
  - bbor 2y ago
    Well put, and I agree with your general approach/skepticism! That said; 1. I don’t think we need to meet “always can improve itself”; rather, the contention is that there’s a high chance that there’s lots of improvement left to be had. An empirical claim rather than a theoretical one, in other words. The simple fact that humans are evolved creatures backs this up in spades, IMO — we’re still improving our own cognition by leaps and bounds using institutions, tools, and methods, and I don’t see any reason why that same dynamic wouldn’t apply to artificial cognitive systems. 2. I think dodging “intelligence” is exactly what he’s trying to do by listing concrete behavioral/cognitive differences. “Intelligence” is pretty much a useless term in science IMO, as was best expounded by Turing in his seminal 1950 paper, Computing Machinery and Intelligence: https://courses.cs.umbc.edu/471/papers/turing.pdf https://courses.cs.umbc.edu/471/papers/turing.pdf People remember that paper as “you can tell a real AI when it can trick you”, but that’s not what he was trying to say at all; rather, he was trying to highlight that there is no such thing as a “real” AI, or “real” thinking, or “real” intelligence — just behavioral similarities and dissimilarities. If the second bit grabs you/anyone, definitely watch some Chomsky lectures on YouTube about cognition. He centers his analysis on this, pejoratively calling discussions about “person”, “intelligence”, “thinking”, etc. mere terminological disputes, not specific enough to have much scientific value. His old refrain is a great one: does an airplane fly? Does a submarine swim? Kinda, if you want!
glhaynes 2y ago
My dumb guy take on it: suppose we build a human-level AGI and it turns out to be limited by compute and memory. Those being limiting factors don’t seem at all far-fetched to me; it seems unlikely that the first real-time AGI will be mostly idling its CPUs. So then wait 18 months and run that same program on a machine that’s this year’s model plus a Moore’s Law doubling. You’ve probably got ASI. Right?
gary_0 2y ago
> I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. ...how on earth would such an AI even know it succeeded? This touches on one of the few good reasons to be less ardent about AI/AGI: "intelligence" is not very well-defined and we don't have very good ways of measuring it. I don't think this is a total blocker, but it might present difficulties. What if our current approach ends up creating super-autism instead of super-intelligence? There's a long history (starting with Asimov) of drilling down into how vague things get when you start trying to draw clean lines around AI and its implications and those questions are yet to be definitively answered. However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. Humans, after all, could barely master fire a few hundred thousand years ago, and now we have an understanding of the universe that the earliest humans were incapable of comprehending on even a basic level. It's obvious to me that simpler things are capable of building more complex things; blind evolution can do it, so there's nothing in physics preventing intelligence bootstrapping. We also have the ability to use intellectual division of labor to build tools that vastly enhance our abilities as a species. The human brain as hardware is far from some impassable apex; hardware can always be used to build better hardware, much like the earliest CPUs were themselves used to design better CPUs.
- AnimalMuppet 2y ago
  > However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. How about this weaker statement: "It is not obviously true that humans (or a human-level AI) can bootstrap to a superhuman AI in a small number of years."
  - gary_0 2y ago
    That one's a definite possibility, I think. I do think GenAI will prove to be very useful, possibly even world-changing (for better or worse), and the current frenzy of investment and research will probably turn up other useful ANN techniques (eventually). But the success of LLMs is not the Final Portent before the Singularity Arrives. I can think of quite a few major lines of research[0] that might be required before we can achieve superintelligence, and it's far from a given they'll succeed (or even get off the ground) any time soon. Especially if the economy loses the appetite for throwing billions of dollars into the AI furnace. [0] My pet theory is that embodiment might be required rather than solely relying on mostly language-based training data. A general intelligence might need to learn via interaction (initially; then you can just copy-paste the weights), because language is merely the hearsay of actual reality. Also, using attention-based "hacks" for easy parallelism to avoid needing exaflops that our hardware doesn't have might also be an issue (that's just a guess though, as I'm no AI expert).
LinuxAmbulance 2y ago
Far too much of anything singularity related depends on "And then magic happens". As if intelligence and / or consciousness arises spontaneously once there's sufficient resources to support it. Which is ludicrous, on the face of it. Let alone bootstrapping intelligence.

bbor 2y ago

If you're ever stuck wondering why a bunch of smart, motivated people with no clear corrupting motivations are being idiotic, that's a strong heuristic that you should spend a bit more time analyzing the issue, IMO ;). "Ugh, why is everyone else so stupid" is a common take for undergrad engineers, but I'm sure you've grown out of it in other ways. Anyway, more substantively: The simple answer is that people have thought about it in depth, most famously noted doomer Eliezer Yudkowsky in Intelligence Explosion Microeconomics (2013)[1] and its main citation, Irving John Good's Speculations Concerning the First Ultraintelligent Machine (1965)[2]. Another common citation that drops a bit of rigour in the name of approachability is Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies[3]. [ETA: to put it even simpler: a system that improves itself is a (the?) quintessential setup for exponential growth. E.g. compound interest] For the time-bound, the most rigorous treatment of your concern among those three is in Section 3 of Yudkowsky's paper, "From AI to Machine Superintelligence". To list the headings briefly: - Increased computational resources - Communication speed - Increased serial depth (i.e. working memory capacity) - Duplicability (i.e. reliability) - Editability (i.e. we know how computers work) - Goal coordination (this is really just communication speed, again) - Improved rationality (i.e. fewer emotions/accidental instincts getting in the way) Let's drop "human" and "superhuman" for a minute, and just talk about "better computers". I'm assuming you're a software engineer. Don't you see how a real software dev replacement program could be an unimaginable gamechanger for software? Working 24/7, enhancing itself, following TDD perfectly every time, and never ever submitting a PR that isn't rigorously documented and reviewed? All of which only gets better over time, as it develops itself? [1] https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [2] https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e423ee-54e0-4eec-aeca-32b73f851af5/content https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e... [3] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... TL;DR May God have mercy on us all.

amanaplanacanal 2y ago
And if somehow we were still able to control such a thing, it's much more likely that the goal would be "make my investors and I as much money as possible" rather than "solve all of humanities problems".
- bbor 2y ago
  Re: “all of humanities problems”, I think that AI will more productively and safely be deployed in personalized, decentralized contexts. AKA Artificial Self-Instantiation for everyone who wants it, rather than a single Rehoboam god machine[1]. Re: capitalism, this is our chance. The world is about to turn upside down. We must strike while the iron is hot. We don’t need to replace capitalism with communism or any other specific thing; we just need to aggressively question all human hierarchies, and get rid of any that can not justify themselves. A just society will come about piece by piece, in this fashion. This is what Chomsky calls “Anarchy”, which is a much more understandable phrasing than what I was taught in high school, ie “no laws ever of any kind” [1] https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U
LegionMammal978 2y ago
To me, the big dubious part is, how do we know that increasing the capacity of a reasoning machine can always increase the quality of its output by an equal factor, even when its task is 'designing the next iteration of a reasoning machine'? It may very well be that the task of 'designing a reasoning machine' has sharply diminishing returns when you throw more resources at it. (E.g., at least so far, any LLM can only have a very incidental impact on designing the next LLM, it's not going to come up with groundbreaking new approaches.) Yudkowsky builds the whole edifice on top of his very particular conception of intelligence, which insists upon itself, but which I think is far from the only explanation of its nature as observed in humans, other animals, LLMs maybe, and so on.
- bbor 2y ago
  Well put, and I agree with your general approach/skepticism! That said; 1. I don’t think we need to meet “always can improve itself”; rather, the contention is that there’s a high chance that there’s lots of improvement left to be had. An empirical claim rather than a theoretical one, in other words. The simple fact that humans are evolved creatures backs this up in spades, IMO — we’re still improving our own cognition by leaps and bounds using institutions, tools, and methods, and I don’t see any reason why that same dynamic wouldn’t apply to artificial cognitive systems. 2. I think dodging “intelligence” is exactly what he’s trying to do by listing concrete behavioral/cognitive differences. “Intelligence” is pretty much a useless term in science IMO, as was best expounded by Turing in his seminal 1950 paper, Computing Machinery and Intelligence: https://courses.cs.umbc.edu/471/papers/turing.pdf https://courses.cs.umbc.edu/471/papers/turing.pdf People remember that paper as “you can tell a real AI when it can trick you”, but that’s not what he was trying to say at all; rather, he was trying to highlight that there is no such thing as a “real” AI, or “real” thinking, or “real” intelligence — just behavioral similarities and dissimilarities. If the second bit grabs you/anyone, definitely watch some Chomsky lectures on YouTube about cognition. He centers his analysis on this, pejoratively calling discussions about “person”, “intelligence”, “thinking”, etc. mere terminological disputes, not specific enough to have much scientific value. His old refrain is a great one: does an airplane fly? Does a submarine swim? Kinda, if you want!

glhaynes 2y ago

My dumb guy take on it: suppose we build a human-level AGI and it turns out to be limited by compute and memory. Those being limiting factors don’t seem at all far-fetched to me; it seems unlikely that the first real-time AGI will be mostly idling its CPUs. So then wait 18 months and run that same program on a machine that’s this year’s model plus a Moore’s Law doubling. You’ve probably got ASI. Right?

gary_0 2y ago

> I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. ...how on earth would such an AI even know it succeeded? This touches on one of the few good reasons to be less ardent about AI/AGI: "intelligence" is not very well-defined and we don't have very good ways of measuring it. I don't think this is a total blocker, but it might present difficulties. What if our current approach ends up creating super-autism instead of super-intelligence? There's a long history (starting with Asimov) of drilling down into how vague things get when you start trying to draw clean lines around AI and its implications and those questions are yet to be definitively answered. However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. Humans, after all, could barely master fire a few hundred thousand years ago, and now we have an understanding of the universe that the earliest humans were incapable of comprehending on even a basic level. It's obvious to me that simpler things are capable of building more complex things; blind evolution can do it, so there's nothing in physics preventing intelligence bootstrapping. We also have the ability to use intellectual division of labor to build tools that vastly enhance our abilities as a species. The human brain as hardware is far from some impassable apex; hardware can always be used to build better hardware, much like the earliest CPUs were themselves used to design better CPUs.

AnimalMuppet 2y ago
> However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. How about this weaker statement: "It is not obviously true that humans (or a human-level AI) can bootstrap to a superhuman AI in a small number of years."
- gary_0 2y ago
  That one's a definite possibility, I think. I do think GenAI will prove to be very useful, possibly even world-changing (for better or worse), and the current frenzy of investment and research will probably turn up other useful ANN techniques (eventually). But the success of LLMs is not the Final Portent before the Singularity Arrives. I can think of quite a few major lines of research[0] that might be required before we can achieve superintelligence, and it's far from a given they'll succeed (or even get off the ground) any time soon. Especially if the economy loses the appetite for throwing billions of dollars into the AI furnace. [0] My pet theory is that embodiment might be required rather than solely relying on mostly language-based training data. A general intelligence might need to learn via interaction (initially; then you can just copy-paste the weights), because language is merely the hearsay of actual reality. Also, using attention-based "hacks" for easy parallelism to avoid needing exaflops that our hardware doesn't have might also be an issue (that's just a guess though, as I'm no AI expert).

LinuxAmbulance 2y ago

Far too much of anything singularity related depends on "And then magic happens". As if intelligence and / or consciousness arises spontaneously once there's sufficient resources to support it. Which is ludicrous, on the face of it. Let alone bootstrapping intelligence.