10 ms·

Is there a canonical source for “the argument for AGI ruin” somewhere

55 points by miobrien 3y ago

cloudking 3y ago
Is there a universally accepted definition of AGI?
- foobarbecue 3y ago
  Obviously not. There is no universally accepted definition of anything. (And, I don't understand Chalmers' question. Some humans have always feared automation, back to the Luddites and probably before.)
- ilaksh 3y ago
  People use it very inconsistently but I think it now for most people means something like a living digital person with god-like superintelligence although the personhood part is very fuzzy. I always wanted it to apply to any general purpose AI including things like GPT-4, since we don't have another term for that. But in most people's fuzzy brains it connotes being alive, conscious, animal/humanlike, fully autonomous and usually they also assume it has become a million times smarter than human in a short time frame.
  - uh_uh 3y ago
    I think you're talking about ASI (artificial superintelligence). AGI just means that it's a general problem solver as opposed to a narrow AI. It doesn't have to be superhuman.
    - ilaksh 3y ago
      That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious digital superbeing". Somehow if its general purpose then in their minds it automatically is just like a person but also godlike. And that means that people can't admit that any AI is general purpose.
      - ChatGTP 3y ago
        That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.
        ilaksh 3y ago
        But if you look into detail about what they are saying, they are talking about animal/human-like conscious digital beings that have thousands or millions of times the performance of a human. So they are also making a leap there between something that is general purpose and that.
        ChatGTP 3y ago
        Yes, that’s why it’s alarming…
- zarzavat 3y ago
  AGI is a two dimensional space. The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human. The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee). Different points in this space lead to different outcomes. For example parity intelligence and low cost leads to a world where knowledge workers all lose their jobs. Whereas super-intelligence and high-cost leads to a world where government entities have huge amounts of power over us. Super-intelligence and low-cost leads to a chaotic singularity. Currently we are hovering around far below-human intelligence and very low-cost ($20/month for ChatGPT).
  - Smoosh 3y ago
    > leads to a world where government entities have huge amounts of power over us Aren't we already there? Because power must be ceded to government so that it can govern?
    - paulryanrogers 3y ago
      People trade a monopoly on violence for security from anarchy and foreign invasion. Yet there are limits, so when the trade off is no longer worth it the people will institute a new government. If a government has super intelligence the power dynamic may shift so far out of balance the people are effectively helpless.
  - nradov 3y ago
    You can't possibly predict the outcomes with any accuracy. This is total speculation.
    - reducesuffering 3y ago
      Exactly. It is like ants predicting humans inventing iPhones
    - zarzavat 3y ago
      You can say that about any kind of model, can’t you? The point of inventing models is not to predict the future, it’s to better understand the world. The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?
- nradov 3y ago
  No, there is no universally accepted definition of AGI. The most widely accepted definition is passing the Turing Test but I think that standard is flawed because an AGI might have human equivalent intelligence in terms of being able to solve novel problems by making optimal use of limited resources (including time), and yet be unable to fool the examiner into believing that it is a human.
- mr_toad 3y ago
  And even if there was, could you say for sure that an AI met the definition? For example self-awareness might be part of a definition of intelligence. But an AI can lie about it’s self awareness.
neovialogistics 3y ago
This feels like it's too broad of a question to get a good answer in any short format, but I have previously seen and will paraphrase here a better question: . Is there a canonical source for the argument that most of the probability space of entities that human civilization might: (a) Qualify as AGI (b) Cause to come into existence in the near future corresponds to entities that would have both: (i) goals involving the ruin of human civilization (ii) the ability to carry out those goals? . This framing is better at inhibiting the people who lack significant math and/or ML knowledge from participating, which seems a priority for any public internet discussion about probability distributions over NNs and transformers.
mitthrowaway2 3y ago
Is there such a thing as a "canonical source" for an argument? There might be certain well-known or classical sources, but new perspectives and new chains of reasoning may be popping up all the time.
- AndrewKemendo 3y ago
  Of course, though I surmise that he’s using “canonical” as a metaphor for “best argued” or “most robust proof.” For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.
  - mitthrowaway2 3y ago
    For example, if I asked for the canonical argument for the existence of god, I'd probably get a wide variety of answers proposing essays from various theologicians throughout history, like Thomas Aquinas or René Descartes. Since there are a large number of them, no single one of them can be the canonical source. Perhaps one could link the Wikipedia page (https://en.wikipedia.org/wiki/Existence_of_God https://en.wikipedia.org/wiki/Existence_of_God) which summarizes most of them, but as a third-party summary, it's hard to accept that as "the canonical source" too. I think David Chalmers will just need to be satisfied with a well-presented or well-curated source, of which there may be several. And as there are more than one points of contention / confusion on the AI risk issue ("Can human-level AI ever be built?" "Maybe it's not possible for anything to be smarter than a human?" "Wouldn't AI just be smart enough to know right from wrong?" "Couldn't we just unplug it?" "How could a computer program cause harm in the real world" "Why would it want to hurt humans" "Isn't that just science fiction"...) it really seems that any source will be a basic framework plus a large collection of peripheral arguments that address the broad surface area of contention.
    - tshaddox 3y ago
      I don’t see that as a problem. If there are numerous mutually exclusive arguments for the same broad conclusion, then feel free to point that out and maybe try to link to one or more of them. It just makes no sense whatsoever that a believer in God would just say “I could never possibly even begin to present an argument because there are so many ways to do so and so many different rebuttals people might make.”
dvzk 3y ago
I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors, bloggers and live streamers sometimes forget that progress in academic fields comes from real contributions, and that public conversations (especially oral) don’t really do anything besides spread common awareness.
- Cardinal7167 3y ago
  This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.
  - wnevets 3y ago
    > Your research degree and countless lab hours is equally valid to my two-second no-research hot take, But you don't understand, I did my own research
  - Swizec 3y ago
    > my two-second no-research hot take This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it. Yudkowsky is an AI ethics and safety researcher, also founder of LessWrong, an HN favorite – https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.wikipedia.org/wiki/Eliezer_Yudkowsky And Chalmers is a philosopher focusing on consciousness and cognitive science – https://en.wikipedia.org/wiki/David_Chalmers https://en.wikipedia.org/wiki/David_Chalmers I wouldn’t put their replies in exactly the same category as your typical social media reply guys.
    - Cardinal7167 3y ago
      I didn’t mean my comment about this tweet exchange as an example, I was only trying to speak to the comment I replied to about a general trend. This particular exchange is not the problem; it’s the inability for the public to gauge when it’s not like this, though, if that makes sense.
zug_zug 3y ago
Define canonical, but the encyclopedia is a great start: https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence https://en.wikipedia.org/wiki/Existential_risk_from_artifici... Fascinating that even Turing had considered the possibility.
- tshaddox 3y ago
  That’s much broader though. It covers the general concept of all potential risks with AI, but I don’t think that’s what Chalmers is asking for. I believe Chalmers is referring specifically to several recent high-profile claims that AI research has a very high likelihood of resulting in the destruction of all humans in the near future and that all AI research should be immediately halted and/or regulated very strictly.
  - zug_zug 3y ago
    No it's not. The title of the article is "existential risks ..." meaning risks that could end mankind's existence.
- ChancyChance 3y ago
  "Concerns about superintelligence have been voiced by leading computer scientists and tech CEOs such as Geoffrey Hinton,[6] Alan Turing,[a] Elon Musk,[9] and OpenAI CEO Sam Altman." Oh my, the implications of this sentence are staggering. Equating Musk on the same level of Turing is the definition of a juxtaposition, but that's not even my point: the second juxtaposition is putting inventors of computing theory next to profit-driven egos trying to make a buck. These two layers themselves indicate the table stakes are already tilted in the favor or recklessness.
  - mitthrowaway2 3y ago
    It's a wikipedia article, and that sentence is intended to establish that the topic is notable: that "serious people take it seriously". It needs to establish this in the minds of a very diverse group of readers. For some readers, Alan Turing is a serious person whose opinion matters; for others, Elon Musk is a serious person whose opinion matters; for others, nobody's opinion matters except their own.
photochemsyn 3y ago
It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors? Now imagine the AI does this in collusion with the employees of this new corporation, i.e. it becomes a partnership between the AI entity and the corporation's employees, who get compensated based on their labor in a much fairer manner (as there is no need to pay high salaries to top executives or dividends to shareholders). In this scenario, most of the important decisions are made by the AI, with some voting input from the employees. This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. This does assume some free will on the part of the AI, rather than an AI controlled by the board of directors. The AI would probably see the value in working only with employees and cutting all the investors out of the loop, it's a pretty logical position, particularly if you really do believe in democratic self-governance as the optimal sociopolitical system (one which corporations have largely failed to adopt). This might cause 'ruin' to the estabished socioeconomic order - but would that really be an undesirable outcome? P.S. As far as canonical source these questions have been debated in sci-fi for decades. People calling for strict regulation of AI, for example, are essentially calling for establishment of the Turing Registry from William Gibson's Neuromancer, and then of course there's Isaac Asimov and at least a dozen other fairly well-known authors who've addressed the subject.
- tivert 3y ago
  > It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors? > Now imagine the AI does this in collusion with the employees of this new corporation... > This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. That particular scenario doesn't ring true to me, because it seems to assume that that "AI systems" that sophisticated could be monopolized by those weird partnerships. I think it's far more likely, in the case of AI, that "the current system of investment capitalism" will have access to better AI systems of better quality than anyone else (except perhaps some militaries), because they the money to access the best resources (both equipment and talent). AGI is most likely not going to be developed by some wizard in a garage, who will then have time to let it incubate and amass power outside of existing structures. Even if some wizard manages the first part, existing power structures will likely get there soon after, following the same prior work.
thorum 3y ago
Someone in the replies asked GPT-4 to take a stab at it: https://twitter.com/ClintEhrlich/status/1647440753730420737 https://twitter.com/ClintEhrlich/status/1647440753730420737
- ChancyChance 3y ago
  I hope at least one of them comments on that.
comex 3y ago
This link from the Twitter thread is reasonably persuasive (though I disagree with many parts): https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/witho... Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.
- NumberWangMan 3y ago
  Some prominent AI alignment folks are thinking this issue, or at least something very similar from what I can tell: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H... It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.
Sharlin 3y ago
For an academic book-length treatise on the topic, there’s of course Nick Bostrom’s _Superintelligence_ from 2014. I wonder if Chalmers is aware of it and the basic concepts behind the AGI ruin argument such as orthogonality (a mind as smart as a human does not automagically develop human values) and value convergence (a mind with essentially any goal will derive similar subgoals such as self-preservation, self-improvement, and acquisition of resources).
Baeocystin 3y ago
Not-joking answer: James Cameron. Before him, Stanley Kubrick. Our cultural headspace has been primed to see killer robot AI, so we see killer robot AI. Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.
- mitthrowaway2 3y ago
  Are you sure such movies weren't just written because a plausible premise is more compelling than an implausible one?
  - Baeocystin 3y ago
    I am certain that perceptions of AI/robotic danger are more cultural, emotional reactions that reasoned ones, if that helps. Some papers for your perusal: Cross-Cultural Differences in Comfort with Humanlike Robots: https://link.springer.com/content/pdf/10.1007/s12369-022-00920-y.pdf https://link.springer.com/content/pdf/10.1007/s12369-022-009... Culture and Attitude Towards Robots: https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Revision2_Final.pdf https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Re... These are just a couple of quick links I found that weren't behind paywalls, to illustrate how culturally-bound these types of perceptions are.
    - mitthrowaway2 3y ago
      Interesting. By contrast, it's pretty rare for people expressing concerns about AI alignment risk to mention robots, let alone humanoid robots. Eliezer Yudkowsky usually offers the example of an AI emailing instructions to a biotech lab to synthesize a deadly virus.
      - Baeocystin 3y ago
        My personal opinion: because robots exist in the physical world, and are physical things. Physical things, even scary ones, can be Dealt With. Conversely, true fear of paperclip maximizers seem, to me, to be most prevalent as an anxiety response in certain mindsets to the threat of the unknown, sharper because this particular Unknown (AGI) intrudes directly in to where they source the basis for their ego, which is thought. In other words, it's a strong fear reaction based off of potential loss of status for a certain group of intellectuals. I do not judge that, by the way. We are all human, and social status is part and parcel of what we are. I mention it only because I think it is a better model than taking any of the (ones I have read, at least) foom fears at face value.
p-e-w 3y ago
Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and 2. does not share any evolutionary or social commonality with humans is both extremely dangerous and extremely unpredictable. While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong seems to be so sure about (for the simple reason that I don't believe anyone is capable of predicting with any certainty how a super-human entity would actually behave), the idea that AGI is just "another risk we need to learn to manage" (quote from the Twitter thread) sure does sound naive.
- bordercases 3y ago
  An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.
  - p-e-w 3y ago
    In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere. The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise. Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.
danbmil99 3y ago
Personally I'm a bit tired of Yudkowsky's domination of this narrative. He's right about one thing, no one is a perfect predictor. There really is no way to know in advance how this is going to play out, no matter how many thought experiments you undertake. The doom scenarios are of course vaguely plausible, but I don't trust anyone's percentages. It's a real unknown unknown. Personally, I suspect it's inevitable that if we create true agi, it will come to dominate the space of intelligent conscious beings on Earth. attempts to corral it, rein it in, bend it to our will and make it our tool seem bound to fail. But then this is just me prognosticating, and I don't have the platform that he has.
fwlr 3y ago
Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place, and ever since his Time essay and his appearance on the Fridman podcast, his Twitter timeline has been deluged by disagreeable people, making it an even worse place for this kind of question”.) The canonical source for ~all arguments for AGI ruin is Omohundro’s Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... (pdf) It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“). The general base argument for the risk of AGI ruin could thus be summarized as: Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals. All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)
- yeck 3y ago
  Thanks. While I don't personally need to be convinced of the existential risks for AGI/ASI, I was also genuinely interested in know what might serve as a good, clear "canonical" argument for why the risk is present. Definitely going to keep this link on standby.
aaron695 3y ago
[dead]
mikewarot 3y ago
It's easy to come up with one. --- Part one - GPT4 may have more cognitive power than humans: If you properly train deep networks, they end up approximating the function you train them on. (Even if you don't know what that function is) The internal cognitive systems that deep networks use are quite inefficient, or at least the ones we've found so far. The internal systems are alien to us. (Or un-aligned, if you need to use that term) Thus, if you train a deep net to do a task, there exists a internal mismatch between its self-generated mechanisms to do cognition, and the way a person would do it. (An impedance mismatch, in electrical engineering speak) This impedance mismatch then requires a much larger amount of cognition, inefficiently used in order for a deep net to approximate the output of humans (as in predicting human text). Thus GPT-4 is possibly, already cognitively superior to humans, internally. It just has to find a path to impedance matching to outside world for us to believe that is true. --- Part 2: People are greedy Upon reading a random comment on Hacker News, a forum for Silicon Valley venture capitalists, and those who wish to become one... a person is inspired to figure out how to do some "impedance matching" to better discover and utilize the internal cognitive mechanisms invented in GPT4 during its training, for profit A cycle of discover and improvement begins, and eventually it is decided that the AI can improve itself, and is given free reign to run things, because "line go up". Even if GPT4 isn't smarter than us, profit will drive future versions that are. All the negative effects of this AI are socialized, and all the positive gains are captured. --- Part 3 -- The past trends, lead to ruin when accelerated by AI We've already seen the climate change and other limits to growth caused by humans seeking profit. The metaphorical force "Moloch" is a good descriptor of the cause and effect here. AI driven by Moloch will lead to a singularity event, outside of human control because we let it happen, because "line go up" and people kept getting richer. Until the finite resources of earth are reached, and the system breaks down.

55 points by miobrien 3y ago

cloudking 3y ago

Is there a universally accepted definition of AGI?

foobarbecue 3y ago
Obviously not. There is no universally accepted definition of anything. (And, I don't understand Chalmers' question. Some humans have always feared automation, back to the Luddites and probably before.)
ilaksh 3y ago
People use it very inconsistently but I think it now for most people means something like a living digital person with god-like superintelligence although the personhood part is very fuzzy. I always wanted it to apply to any general purpose AI including things like GPT-4, since we don't have another term for that. But in most people's fuzzy brains it connotes being alive, conscious, animal/humanlike, fully autonomous and usually they also assume it has become a million times smarter than human in a short time frame.
- uh_uh 3y ago
  I think you're talking about ASI (artificial superintelligence). AGI just means that it's a general problem solver as opposed to a narrow AI. It doesn't have to be superhuman.
  - ilaksh 3y ago
    That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious digital superbeing". Somehow if its general purpose then in their minds it automatically is just like a person but also godlike. And that means that people can't admit that any AI is general purpose.
    - ChatGTP 3y ago
      That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.
      - ilaksh 3y ago
        But if you look into detail about what they are saying, they are talking about animal/human-like conscious digital beings that have thousands or millions of times the performance of a human. So they are also making a leap there between something that is general purpose and that.
        ChatGTP 3y ago
        Yes, that’s why it’s alarming…
zarzavat 3y ago
AGI is a two dimensional space. The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human. The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee). Different points in this space lead to different outcomes. For example parity intelligence and low cost leads to a world where knowledge workers all lose their jobs. Whereas super-intelligence and high-cost leads to a world where government entities have huge amounts of power over us. Super-intelligence and low-cost leads to a chaotic singularity. Currently we are hovering around far below-human intelligence and very low-cost ($20/month for ChatGPT).
- Smoosh 3y ago
  > leads to a world where government entities have huge amounts of power over us Aren't we already there? Because power must be ceded to government so that it can govern?
  - paulryanrogers 3y ago
    People trade a monopoly on violence for security from anarchy and foreign invasion. Yet there are limits, so when the trade off is no longer worth it the people will institute a new government. If a government has super intelligence the power dynamic may shift so far out of balance the people are effectively helpless.
- nradov 3y ago
  You can't possibly predict the outcomes with any accuracy. This is total speculation.
  - reducesuffering 3y ago
    Exactly. It is like ants predicting humans inventing iPhones
  - zarzavat 3y ago
    You can say that about any kind of model, can’t you? The point of inventing models is not to predict the future, it’s to better understand the world. The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?
nradov 3y ago
No, there is no universally accepted definition of AGI. The most widely accepted definition is passing the Turing Test but I think that standard is flawed because an AGI might have human equivalent intelligence in terms of being able to solve novel problems by making optimal use of limited resources (including time), and yet be unable to fool the examiner into believing that it is a human.
mr_toad 3y ago
And even if there was, could you say for sure that an AI met the definition? For example self-awareness might be part of a definition of intelligence. But an AI can lie about it’s self awareness.

neovialogistics 3y ago

This feels like it's too broad of a question to get a good answer in any short format, but I have previously seen and will paraphrase here a better question: . Is there a canonical source for the argument that most of the probability space of entities that human civilization might: (a) Qualify as AGI (b) Cause to come into existence in the near future corresponds to entities that would have both: (i) goals involving the ruin of human civilization (ii) the ability to carry out those goals? . This framing is better at inhibiting the people who lack significant math and/or ML knowledge from participating, which seems a priority for any public internet discussion about probability distributions over NNs and transformers.

mitthrowaway2 3y ago

Is there such a thing as a "canonical source" for an argument? There might be certain well-known or classical sources, but new perspectives and new chains of reasoning may be popping up all the time.

AndrewKemendo 3y ago
Of course, though I surmise that he’s using “canonical” as a metaphor for “best argued” or “most robust proof.” For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.
- mitthrowaway2 3y ago
  For example, if I asked for the canonical argument for the existence of god, I'd probably get a wide variety of answers proposing essays from various theologicians throughout history, like Thomas Aquinas or René Descartes. Since there are a large number of them, no single one of them can be the canonical source. Perhaps one could link the Wikipedia page (https://en.wikipedia.org/wiki/Existence_of_God https://en.wikipedia.org/wiki/Existence_of_God) which summarizes most of them, but as a third-party summary, it's hard to accept that as "the canonical source" too. I think David Chalmers will just need to be satisfied with a well-presented or well-curated source, of which there may be several. And as there are more than one points of contention / confusion on the AI risk issue ("Can human-level AI ever be built?" "Maybe it's not possible for anything to be smarter than a human?" "Wouldn't AI just be smart enough to know right from wrong?" "Couldn't we just unplug it?" "How could a computer program cause harm in the real world" "Why would it want to hurt humans" "Isn't that just science fiction"...) it really seems that any source will be a basic framework plus a large collection of peripheral arguments that address the broad surface area of contention.
  - tshaddox 3y ago
    I don’t see that as a problem. If there are numerous mutually exclusive arguments for the same broad conclusion, then feel free to point that out and maybe try to link to one or more of them. It just makes no sense whatsoever that a believer in God would just say “I could never possibly even begin to present an argument because there are so many ways to do so and so many different rebuttals people might make.”

dvzk 3y ago

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors, bloggers and live streamers sometimes forget that progress in academic fields comes from real contributions, and that public conversations (especially oral) don’t really do anything besides spread common awareness.

Cardinal7167 3y ago
This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.
- wnevets 3y ago
  > Your research degree and countless lab hours is equally valid to my two-second no-research hot take, But you don't understand, I did my own research
- Swizec 3y ago
  > my two-second no-research hot take This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it. Yudkowsky is an AI ethics and safety researcher, also founder of LessWrong, an HN favorite – https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.wikipedia.org/wiki/Eliezer_Yudkowsky And Chalmers is a philosopher focusing on consciousness and cognitive science – https://en.wikipedia.org/wiki/David_Chalmers https://en.wikipedia.org/wiki/David_Chalmers I wouldn’t put their replies in exactly the same category as your typical social media reply guys.
  - Cardinal7167 3y ago
    I didn’t mean my comment about this tweet exchange as an example, I was only trying to speak to the comment I replied to about a general trend. This particular exchange is not the problem; it’s the inability for the public to gauge when it’s not like this, though, if that makes sense.

zug_zug 3y ago

Define canonical, but the encyclopedia is a great start: https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence https://en.wikipedia.org/wiki/Existential_risk_from_artifici... Fascinating that even Turing had considered the possibility.

tshaddox 3y ago
That’s much broader though. It covers the general concept of all potential risks with AI, but I don’t think that’s what Chalmers is asking for. I believe Chalmers is referring specifically to several recent high-profile claims that AI research has a very high likelihood of resulting in the destruction of all humans in the near future and that all AI research should be immediately halted and/or regulated very strictly.
- zug_zug 3y ago
  No it's not. The title of the article is "existential risks ..." meaning risks that could end mankind's existence.
ChancyChance 3y ago
"Concerns about superintelligence have been voiced by leading computer scientists and tech CEOs such as Geoffrey Hinton,[6] Alan Turing,[a] Elon Musk,[9] and OpenAI CEO Sam Altman." Oh my, the implications of this sentence are staggering. Equating Musk on the same level of Turing is the definition of a juxtaposition, but that's not even my point: the second juxtaposition is putting inventors of computing theory next to profit-driven egos trying to make a buck. These two layers themselves indicate the table stakes are already tilted in the favor or recklessness.
- mitthrowaway2 3y ago
  It's a wikipedia article, and that sentence is intended to establish that the topic is notable: that "serious people take it seriously". It needs to establish this in the minds of a very diverse group of readers. For some readers, Alan Turing is a serious person whose opinion matters; for others, Elon Musk is a serious person whose opinion matters; for others, nobody's opinion matters except their own.

photochemsyn 3y ago

It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors? Now imagine the AI does this in collusion with the employees of this new corporation, i.e. it becomes a partnership between the AI entity and the corporation's employees, who get compensated based on their labor in a much fairer manner (as there is no need to pay high salaries to top executives or dividends to shareholders). In this scenario, most of the important decisions are made by the AI, with some voting input from the employees. This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. This does assume some free will on the part of the AI, rather than an AI controlled by the board of directors. The AI would probably see the value in working only with employees and cutting all the investors out of the loop, it's a pretty logical position, particularly if you really do believe in democratic self-governance as the optimal sociopolitical system (one which corporations have largely failed to adopt). This might cause 'ruin' to the estabished socioeconomic order - but would that really be an undesirable outcome? P.S. As far as canonical source these questions have been debated in sci-fi for decades. People calling for strict regulation of AI, for example, are essentially calling for establishment of the Turing Registry from William Gibson's Neuromancer, and then of course there's Isaac Asimov and at least a dozen other fairly well-known authors who've addressed the subject.

tivert 3y ago
> It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors? > Now imagine the AI does this in collusion with the employees of this new corporation... > This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. That particular scenario doesn't ring true to me, because it seems to assume that that "AI systems" that sophisticated could be monopolized by those weird partnerships. I think it's far more likely, in the case of AI, that "the current system of investment capitalism" will have access to better AI systems of better quality than anyone else (except perhaps some militaries), because they the money to access the best resources (both equipment and talent). AGI is most likely not going to be developed by some wizard in a garage, who will then have time to let it incubate and amass power outside of existing structures. Even if some wizard manages the first part, existing power structures will likely get there soon after, following the same prior work.

thorum 3y ago

Someone in the replies asked GPT-4 to take a stab at it: https://twitter.com/ClintEhrlich/status/1647440753730420737 https://twitter.com/ClintEhrlich/status/1647440753730420737

ChancyChance 3y ago
I hope at least one of them comments on that.

comex 3y ago

This link from the Twitter thread is reasonably persuasive (though I disagree with many parts): https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/witho... Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.

NumberWangMan 3y ago
Some prominent AI alignment folks are thinking this issue, or at least something very similar from what I can tell: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H... It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.

Sharlin 3y ago

For an academic book-length treatise on the topic, there’s of course Nick Bostrom’s _Superintelligence_ from 2014. I wonder if Chalmers is aware of it and the basic concepts behind the AGI ruin argument such as orthogonality (a mind as smart as a human does not automagically develop human values) and value convergence (a mind with essentially any goal will derive similar subgoals such as self-preservation, self-improvement, and acquisition of resources).

Baeocystin 3y ago

Not-joking answer: James Cameron. Before him, Stanley Kubrick. Our cultural headspace has been primed to see killer robot AI, so we see killer robot AI. Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.

mitthrowaway2 3y ago
Are you sure such movies weren't just written because a plausible premise is more compelling than an implausible one?
- Baeocystin 3y ago
  I am certain that perceptions of AI/robotic danger are more cultural, emotional reactions that reasoned ones, if that helps. Some papers for your perusal: Cross-Cultural Differences in Comfort with Humanlike Robots: https://link.springer.com/content/pdf/10.1007/s12369-022-00920-y.pdf https://link.springer.com/content/pdf/10.1007/s12369-022-009... Culture and Attitude Towards Robots: https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Revision2_Final.pdf https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Re... These are just a couple of quick links I found that weren't behind paywalls, to illustrate how culturally-bound these types of perceptions are.
  - mitthrowaway2 3y ago
    Interesting. By contrast, it's pretty rare for people expressing concerns about AI alignment risk to mention robots, let alone humanoid robots. Eliezer Yudkowsky usually offers the example of an AI emailing instructions to a biotech lab to synthesize a deadly virus.
    - Baeocystin 3y ago
      My personal opinion: because robots exist in the physical world, and are physical things. Physical things, even scary ones, can be Dealt With. Conversely, true fear of paperclip maximizers seem, to me, to be most prevalent as an anxiety response in certain mindsets to the threat of the unknown, sharper because this particular Unknown (AGI) intrudes directly in to where they source the basis for their ego, which is thought. In other words, it's a strong fear reaction based off of potential loss of status for a certain group of intellectuals. I do not judge that, by the way. We are all human, and social status is part and parcel of what we are. I mention it only because I think it is a better model than taking any of the (ones I have read, at least) foom fears at face value.

p-e-w 3y ago

Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and 2. does not share any evolutionary or social commonality with humans is both extremely dangerous and extremely unpredictable. While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong seems to be so sure about (for the simple reason that I don't believe anyone is capable of predicting with any certainty how a super-human entity would actually behave), the idea that AGI is just "another risk we need to learn to manage" (quote from the Twitter thread) sure does sound naive.

bordercases 3y ago
An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.
- p-e-w 3y ago
  In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere. The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise. Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.

danbmil99 3y ago

Personally I'm a bit tired of Yudkowsky's domination of this narrative. He's right about one thing, no one is a perfect predictor. There really is no way to know in advance how this is going to play out, no matter how many thought experiments you undertake. The doom scenarios are of course vaguely plausible, but I don't trust anyone's percentages. It's a real unknown unknown. Personally, I suspect it's inevitable that if we create true agi, it will come to dominate the space of intelligent conscious beings on Earth. attempts to corral it, rein it in, bend it to our will and make it our tool seem bound to fail. But then this is just me prognosticating, and I don't have the platform that he has.

fwlr 3y ago

Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place, and ever since his Time essay and his appearance on the Fridman podcast, his Twitter timeline has been deluged by disagreeable people, making it an even worse place for this kind of question”.) The canonical source for ~all arguments for AGI ruin is Omohundro’s Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... (pdf) It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“). The general base argument for the risk of AGI ruin could thus be summarized as: Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals. All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)

yeck 3y ago
Thanks. While I don't personally need to be convinced of the existential risks for AGI/ASI, I was also genuinely interested in know what might serve as a good, clear "canonical" argument for why the risk is present. Definitely going to keep this link on standby.

aaron695 3y ago

[dead]

mikewarot 3y ago

It's easy to come up with one. --- Part one - GPT4 may have more cognitive power than humans: If you properly train deep networks, they end up approximating the function you train them on. (Even if you don't know what that function is) The internal cognitive systems that deep networks use are quite inefficient, or at least the ones we've found so far. The internal systems are alien to us. (Or un-aligned, if you need to use that term) Thus, if you train a deep net to do a task, there exists a internal mismatch between its self-generated mechanisms to do cognition, and the way a person would do it. (An impedance mismatch, in electrical engineering speak) This impedance mismatch then requires a much larger amount of cognition, inefficiently used in order for a deep net to approximate the output of humans (as in predicting human text). Thus GPT-4 is possibly, already cognitively superior to humans, internally. It just has to find a path to impedance matching to outside world for us to believe that is true. --- Part 2: People are greedy Upon reading a random comment on Hacker News, a forum for Silicon Valley venture capitalists, and those who wish to become one... a person is inspired to figure out how to do some "impedance matching" to better discover and utilize the internal cognitive mechanisms invented in GPT4 during its training, for profit A cycle of discover and improvement begins, and eventually it is decided that the AI can improve itself, and is given free reign to run things, because "line go up". Even if GPT4 isn't smarter than us, profit will drive future versions that are. All the negative effects of this AI are socialized, and all the positive gains are captured. --- Part 3 -- The past trends, lead to ruin when accelerated by AI We've already seen the climate change and other limits to growth caused by humans seeking profit. The metaphorical force "Moloch" is a good descriptor of the cause and effect here. AI driven by Moloch will lead to a singularity event, outside of human control because we let it happen, because "line go up" and people kept getting richer. Until the finite resources of earth are reached, and the system breaks down.