3 ms·

If you're ever stuck wondering why a bunch of smart, motivated people with no clear corrupting motivations are being idiotic, that's a strong heuristic that you

by bbor 2y ago

If you're ever stuck wondering why a bunch of smart, motivated people with no clear corrupting motivations are being idiotic, that's a strong heuristic that you should spend a bit more time analyzing the issue, IMO ;). "Ugh, why is everyone else so stupid" is a common take for undergrad engineers, but I'm sure you've grown out of it in other ways. Anyway, more substantively: The simple answer is that people have thought about it in depth, most famously noted doomer Eliezer Yudkowsky in Intelligence Explosion Microeconomics (2013)[1] and its main citation, Irving John Good's Speculations Concerning the First Ultraintelligent Machine (1965)[2]. Another common citation that drops a bit of rigour in the name of approachability is Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies[3]. [ETA: to put it even simpler: a system that improves itself is a (the?) quintessential setup for exponential growth. E.g. compound interest] For the time-bound, the most rigorous treatment of your concern among those three is in Section 3 of Yudkowsky's paper, "From AI to Machine Superintelligence". To list the headings briefly: - Increased computational resources - Communication speed - Increased serial depth (i.e. working memory capacity) - Duplicability (i.e. reliability) - Editability (i.e. we know how computers work) - Goal coordination (this is really just communication speed, again) - Improved rationality (i.e. fewer emotions/accidental instincts getting in the way) Let's drop "human" and "superhuman" for a minute, and just talk about "better computers". I'm assuming you're a software engineer. Don't you see how a real software dev replacement program could be an unimaginable gamechanger for software? Working 24/7, enhancing itself, following TDD perfectly every time, and never ever submitting a PR that isn't rigorously documented and reviewed? All of which only gets better over time, as it develops itself? [1] https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [2] https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e423ee-54e0-4eec-aeca-32b73f851af5/content https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e... [3] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... TL;DR May God have mercy on us all.

amanaplanacanal 2y ago
And if somehow we were still able to control such a thing, it's much more likely that the goal would be "make my investors and I as much money as possible" rather than "solve all of humanities problems".
- bbor 2y ago
  Re: “all of humanities problems”, I think that AI will more productively and safely be deployed in personalized, decentralized contexts. AKA Artificial Self-Instantiation for everyone who wants it, rather than a single Rehoboam god machine[1]. Re: capitalism, this is our chance. The world is about to turn upside down. We must strike while the iron is hot. We don’t need to replace capitalism with communism or any other specific thing; we just need to aggressively question all human hierarchies, and get rid of any that can not justify themselves. A just society will come about piece by piece, in this fashion. This is what Chomsky calls “Anarchy”, which is a much more understandable phrasing than what I was taught in high school, ie “no laws ever of any kind” [1] https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U
LegionMammal978 2y ago
To me, the big dubious part is, how do we know that increasing the capacity of a reasoning machine can always increase the quality of its output by an equal factor, even when its task is 'designing the next iteration of a reasoning machine'? It may very well be that the task of 'designing a reasoning machine' has sharply diminishing returns when you throw more resources at it. (E.g., at least so far, any LLM can only have a very incidental impact on designing the next LLM, it's not going to come up with groundbreaking new approaches.) Yudkowsky builds the whole edifice on top of his very particular conception of intelligence, which insists upon itself, but which I think is far from the only explanation of its nature as observed in humans, other animals, LLMs maybe, and so on.
- bbor 2y ago
  Well put, and I agree with your general approach/skepticism! That said; 1. I don’t think we need to meet “always can improve itself”; rather, the contention is that there’s a high chance that there’s lots of improvement left to be had. An empirical claim rather than a theoretical one, in other words. The simple fact that humans are evolved creatures backs this up in spades, IMO — we’re still improving our own cognition by leaps and bounds using institutions, tools, and methods, and I don’t see any reason why that same dynamic wouldn’t apply to artificial cognitive systems. 2. I think dodging “intelligence” is exactly what he’s trying to do by listing concrete behavioral/cognitive differences. “Intelligence” is pretty much a useless term in science IMO, as was best expounded by Turing in his seminal 1950 paper, Computing Machinery and Intelligence: https://courses.cs.umbc.edu/471/papers/turing.pdf https://courses.cs.umbc.edu/471/papers/turing.pdf People remember that paper as “you can tell a real AI when it can trick you”, but that’s not what he was trying to say at all; rather, he was trying to highlight that there is no such thing as a “real” AI, or “real” thinking, or “real” intelligence — just behavioral similarities and dissimilarities. If the second bit grabs you/anyone, definitely watch some Chomsky lectures on YouTube about cognition. He centers his analysis on this, pejoratively calling discussions about “person”, “intelligence”, “thinking”, etc. mere terminological disputes, not specific enough to have much scientific value. His old refrain is a great one: does an airplane fly? Does a submarine swim? Kinda, if you want!
staticman2 2y ago
The objection above was "I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. It is bizarre that this claim is accepted with "little doubt" when there are very good reasons to doubt it: how on earth would such an AI even know it succeeded? How would it define the goal?" In response Yudkowsky is presented as a sort of superintelligent human with the answers, but of course we're back to the original problem, how do we know Yudowsky, who as far as I know does not make disprovable claims, is a superintelligent human with the answers? How would we define the goal?

by bbor 2y ago

amanaplanacanal 2y ago

And if somehow we were still able to control such a thing, it's much more likely that the goal would be "make my investors and I as much money as possible" rather than "solve all of humanities problems".

bbor 2y ago
Re: “all of humanities problems”, I think that AI will more productively and safely be deployed in personalized, decentralized contexts. AKA Artificial Self-Instantiation for everyone who wants it, rather than a single Rehoboam god machine[1]. Re: capitalism, this is our chance. The world is about to turn upside down. We must strike while the iron is hot. We don’t need to replace capitalism with communism or any other specific thing; we just need to aggressively question all human hierarchies, and get rid of any that can not justify themselves. A just society will come about piece by piece, in this fashion. This is what Chomsky calls “Anarchy”, which is a much more understandable phrasing than what I was taught in high school, ie “no laws ever of any kind” [1] https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U https://youtu.be/SSRZfDL4874?si=XQzmR2V5SwiiiX3U

LegionMammal978 2y ago

To me, the big dubious part is, how do we know that increasing the capacity of a reasoning machine can always increase the quality of its output by an equal factor, even when its task is 'designing the next iteration of a reasoning machine'? It may very well be that the task of 'designing a reasoning machine' has sharply diminishing returns when you throw more resources at it. (E.g., at least so far, any LLM can only have a very incidental impact on designing the next LLM, it's not going to come up with groundbreaking new approaches.) Yudkowsky builds the whole edifice on top of his very particular conception of intelligence, which insists upon itself, but which I think is far from the only explanation of its nature as observed in humans, other animals, LLMs maybe, and so on.

bbor 2y ago
Well put, and I agree with your general approach/skepticism! That said; 1. I don’t think we need to meet “always can improve itself”; rather, the contention is that there’s a high chance that there’s lots of improvement left to be had. An empirical claim rather than a theoretical one, in other words. The simple fact that humans are evolved creatures backs this up in spades, IMO — we’re still improving our own cognition by leaps and bounds using institutions, tools, and methods, and I don’t see any reason why that same dynamic wouldn’t apply to artificial cognitive systems. 2. I think dodging “intelligence” is exactly what he’s trying to do by listing concrete behavioral/cognitive differences. “Intelligence” is pretty much a useless term in science IMO, as was best expounded by Turing in his seminal 1950 paper, Computing Machinery and Intelligence: https://courses.cs.umbc.edu/471/papers/turing.pdf https://courses.cs.umbc.edu/471/papers/turing.pdf People remember that paper as “you can tell a real AI when it can trick you”, but that’s not what he was trying to say at all; rather, he was trying to highlight that there is no such thing as a “real” AI, or “real” thinking, or “real” intelligence — just behavioral similarities and dissimilarities. If the second bit grabs you/anyone, definitely watch some Chomsky lectures on YouTube about cognition. He centers his analysis on this, pejoratively calling discussions about “person”, “intelligence”, “thinking”, etc. mere terminological disputes, not specific enough to have much scientific value. His old refrain is a great one: does an airplane fly? Does a submarine swim? Kinda, if you want!

staticman2 2y ago

The objection above was "I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. It is bizarre that this claim is accepted with "little doubt" when there are very good reasons to doubt it: how on earth would such an AI even know it succeeded? How would it define the goal?" In response Yudkowsky is presented as a sort of superintelligent human with the answers, but of course we're back to the original problem, how do we know Yudowsky, who as far as I know does not make disprovable claims, is a superintelligent human with the answers? How would we define the goal?