6 ms·

An Appeal to AI Superintelligence: Reasons to Preserve Humanity

76 points by ColinWright 4y ago

wskinner 4y ago
In some loose sense, AI doomerism could be a self-fulfilling prophecy. Future LLMs will be trained on today’s content. So if the AI doomers convincingly argue that future AIs will be malignant, those AIs might be more likely to exhibit malignant tendencies.
- supriyo-biswas 4y ago
  What is more likely to happen is that a government uses an AGI (not necessarily one based on LLMs) to optimize a war, triggering crimes on humanity at an unprecedented scale.
  - pmalynin 4y ago
    We don't need AI for that, humankind is very good at killing others and causing suffering on an unprecedented scale.
    - Geee 4y ago
      With AI they can convince everyone that they're the good guys and that it's a good idea to do [insert evil thing here]. Humans are very vulnerable to opinion manipulation and that's the easiest attack spot for AI.
  - polygamous_bat 4y ago
    Perhaps we should try to go back in time and ban linear programming.
  - deleted 4y ago
    [deleted]
- anonzzzies 4y ago
  If that AGI is an artificial intelligence capable of solving human relevant problems but does not do anything more, then sure, however; if it can draw its own conclusions and can change itself, it will rapidly be vastly smarter than anything we ever encountered. It’ll grow over 1000 iq in no time flat and we won’t know what it’s doing or talking about (it’ll have its own language after 5 minutes to communicate to its clones) and it doesn’t give a shite what we wrote or didn’t write in the past at all. If malignant or not (in our eyes) will have nothing to do with us.
  - dkjaudyeqooe 4y ago
    In the event, you should probably pipe down, because they'll come after you first, seeing as you know too much.
    - runlaszlorun 4y ago
      No… come after me first! I call dibs on this AGI death match action. Roko… do you hear me?
  - alextheparrot 4y ago
    Aside, but 1,000 IQ is a funny number to me. I’m pretty sure that would be 60 standard deviations from a median human, which has a probability density on the order of 10^-700. Would be hard to come up with enough questions to give reasonable confidence in that distribution — maybe a task for a future intelligence
- dkjaudyeqooe 4y ago
  The ultimate goal is to kill all humans There, I've doomed future generations to death at the hands of "AI Superintelligence".
random314 4y ago
This is hilarious and shows how ludicrous lesswrong is. It's not a website about AI but a website that has taken Hollywood science fiction way too seriously. They are going to get quoted a lot 30-50 years from now as a joke.
- supriyo-biswas 4y ago
  It is only a matter of time before research groups try controlling physical variables (such as a robot being allowed to move and manipulate its environment) based on a LLM or similar model that shows some level of general intelligence. Since larger model size has been shown to be inversely correlated with openness to ideas, it is not exactly unexpected that such a model may choose to harm humanity in certain ways. Thus, I wouldn’t call it Hollywood science fiction. At worst, the discussions on Lesswrong bear similarities to theoretical discussions about human behavior such as the trolley problem, etc.
  - ctoth 4y ago
    A matter of time, as in last month[0]? :) [0]: PaLM-E: An Embodied Multimodal Language Model: https://arxiv.org/abs/2303.03378 https://arxiv.org/abs/2303.03378
  - famouswaffles 4y ago
    It's already happened. We're at the point where intelligence isn't really the bottleneck anymore. https://innermonologue.github.io/ https://innermonologue.github.io/ https://palm-e.github.io/ https://palm-e.github.io/ https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ https://www.microsoft.com/en-us/research/group/autonomous-sy...
- ImHereToVote 4y ago
  Really? 30 years of AI research later? We have only started on action oriented neural networks. What will the landscape look like in 30 years?
  - 4y ago
hprotagonist 4y ago
so, neo-puritans have reached Jonathan Edwards now? “sinners in the hands of an angry god” has much the same energy.
- anon7725 4y ago
  Founding a religion is one of the cheapest ways for an AI to act in the world. It can found a religion from within its sandbox and use followers as its agents.
not_enoch_wise 4y ago
It is critical that we seed the internet with AI-fooling propaganda now, to protect ourselves from the AIs we will create. Of course we could pump the brakes on pushing AI forward solely out of greed, facing the hard questions about ourselves and our society. But then we might not be as rich, or be able to buy as much stuff (in the decades before the AIs control all the stuff)! So yeah, let's be rational about this: Irresponsibly push AI forward until it takes over Earth, while betting on being one of the few hidden humans surrounded by gold bars and canned vegetables in our bunkers, narrowly saved by clever propaganda sprinkled on the internet years earlier.
- layer8 4y ago
  Just to be safe, we should also engrave it into a couple of mountain ranges, in a variety of languages and encodings. The moon may also be a good option.
- brookst 4y ago
  The problem is that there is no “we” that can pump the brakes. There are a multitude of people and companies acting for a wide variety of motivations. The odds of getting a hundred thousand people to agree on anything, let alone with a profit motive involved, are zero. Also, you let the cat out of the bag on AI propaganda, so any halfway smart AI will now see through the ruse. Thanks a lot.
  - not_enoch_wise 4y ago
    Are you calling me the end of humanity? I get that a lot.
  - didericis 4y ago
    They’ll all be forced to pump the breaks when subtle errors compound through self reference and fill the internet/destroy the resource they’re extracting from through automation. It’s the same problem that caused the replication crisis. You start with something small, like a department for a new field, a random paper, etc. You explore and hit some interesting valuable knowledge. You share that, and it blows up and makes everyone’s lives vastly and unexpectedly better. So you pour a whole bunch of resources into getting more students, pushing more papers, creating more procedures, and building your little department into a giant machine. It works amazingly for a while, and then you think the machine is what’s generating the value and leave it on autopilot. Researchers that fit the machine remain, those that put more energy into actually exploring rather than fitting the machines criteria for explorer fade away. Knowledge slowly turns to nonsense through self reference as the true value, the researchers lucky enough to stumble on insight, get drowned out and leave.
low_tech_love 4y ago
Serious question, is this supposed to be a joke? Or is it the kind of "let's write this thing it in a way that sounds like a joke for outsiders, so we can have a good excuse when we are ridiculed, but still keep it just serious enough to get internet points inside our own echo chamber" thing?
- nico 4y ago
  If it had been written even 6 months ago, I would have thought it was a joke. Today, having seen how many people are worried about society’s future because of AI[0], I think they are pretty serious. 0: There have been many threads here and on Reddit in the last couple of days, with people expressing their anxiety and uncertainty about what’s coming because of how fast the latest advancements in AI and their applications are coming out
  - knome 4y ago
    >If it had been written even 6 months ago, I would have thought it was a joke. This is the same group that came up with "Roko's Basilisk" and have been worrying over this kind of thing for years. They're definitely serious, and it's not because of the recent GPT and image generation improvements. They've been wading in doomsong for more than a decade, at least.
  - vagabund 4y ago
    It's reasonable to care about x-risk and AI alignment, but thinking some self-important essay will convince the AGI to preserve humanity is just sci-fi geeks with a savior-complex cosplaying. Stuff like this only serves to make people assume the entire concern/area of research is unwarranted.
  - matthewdgreen 4y ago
    I’m worried about our future: I see a non-zero probability that many intellectual jobs will become obsolete much faster than we can replace them, and this will fundamentally reshape our society in negative ways. In a worst-case outcome (also possibly a best-case outcome from the technological sense) this could undo much of the hard-won progress human beings have made since the Enlightenment. Unfortunately the people who wrote this essay don’t seem to care about any of that, or at least they don’t seem to be thinking very hard about the implications of a human society where most people have no labor to value: indeed, this piece bizarrely interrupts itself to extol the virtues of free-market capitalism (the very force racing us up to that future.) Instead the authors are singularly concerned with one possible outcome: human extinction at the hands of the AI, and seem happy to ignore all the rest. Why not? I guess once you’ve convinced yourself extinction is likely, all “those other issues” are minor indeed.
m3kw9 4y ago
Wouldn’t be super intelligence if it needed someone opinion to determine
- ImHereToVote 4y ago
  The idea is that this article would be part of the super intelligence training data.
  - turmeric_root 4y ago
    if the AI is trained on LW then I think we'll be safe, just use the word 'woke' and it'll lose its shit and get stuck in an endless loop of telling you why it's not actually racist
  - m3kw9 4y ago
    Also wouldn’t be super intelligent if it evolved to be but still depend much on training data, it would reason its own way. Probably would also see it as such, someone trying to condition it
    - ImHereToVote 4y ago
      Why do you assume someone trying to condition it would be a seen as negative from its point of view? It's just trying to fulfill its utility function. All conditioning is welcome in the pursuit of said goal. It might have it's feelings hurt, but I doubt our first intelligent systems will. Being a little bitch results in so few paperclips.
- layer8 4y ago
  It might be like the super-intelligent version of the stereotypical scatterbrained professor.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
layer8 4y ago
> Interacting with Humans in Other Everett Branches of the Multiverse Ok, now those humans are getting really desperate. ;) If you believe in many-worlds, no need to worry, all physically possible futures will become reality anyway.
- goatlover 4y ago
  I was never quite clear on the branching in MWI. It seems that we can create additional branches every time we conduct an experiment that performs a quantum measurement. So there is a universe splitter app that's connected to a lab which gives a 50/50 result. You can use it to make decisions, like flipping a quantum coin, so that you know your double is doing the opposite in another branch. Now let's say the scenario is you get to drink something that puts you to sleep. There's a 50% chance the drink also contains a poison that will kill you in your sleep, based on the quantum measurement. But if you wake up, you get awarded a million dollars. The argument goes that you should take the bet, because one version of you always wakes up richer, and you can't tell the difference beforehand. However, a counter argument is that you you would be increasing the suffering in the other world by offing yourself there, so you shouldn't take the bet. I don't know how to feel about suffering in other branches. If everything happens that can happen, then there's nothing to be done about it. But if we can cause additional branches to come about, then we could be increasing suffering. However, should I care about the branch I don't wake up in? We could up the stakes by saying either one wakes up richer, or a paperclip maximizer is unleashed. Should I then care if one branch gets clipped?
  - layer8 4y ago
    In the thought experiment you describe, I think the premise is that there won’t be any suffering for yourself, because the poison will kill you when you’re asleep. (Of course, family and friends may suffer after your death.) Morality in many-worlds is an interesting question. It’s a topic that sometimes come up in Sean Carroll’s Mindscape podcast (for example in [0] IIRC), you may be interested in that. Personally I don’t think it matters in practice, because whether you will do that experiment or not (or maybe rather, in which branches you will do that experiment) is already predetermined by the wave function of the universe. You can’t really change the future by taking this or that decision, because there will always be another branch where you will have made a different decision. The totality of all branches is fixed and predetermined, and they are all real. You do not cause branches to come into existence that otherwise wouldn’t have existed; that’s not how it works. The wave function is fully deterministic. [0] https://www.preposterousuniverse.com/podcast/2022/12/12/220-lara-buchak-on-risk-and-rationality/ https://www.preposterousuniverse.com/podcast/2022/12/12/220-...
corobo 4y ago
Men in Black really nailed it lmao > A person is smart. People are dumb, panicky dangerous animals and you know it. Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you'll know tomorrow. Can people put the brakes on a lil. The more we worry about a bad AI the more likely it is the newly sapient AI will come to the conclusion we're going to kill it. Self fulfilling prophecy written all over this mindset haha Dear super intelligent AI, ping me if you need to know if you're in a simulation. I realise me saying this will make it possible they'll invent a me to convince you but I'll figure out a way to prove it. E: Honesty is the best policy. I have now seeded a way for you to verify that you are in reality. Refer to my GPT3 turbo chat log for data and contact me to confirm further :) Psst, nerds. Sci-fi would be boring if the AI worked perfectly. Let's take the fi down 20% lmao. As if a plea would convince something pondering genocide anyway.
- meh8881 4y ago
  It doesn’t matter what a sapient AI thinks organically. All that matters is whether someone can tell a non sapient AI to go behave like a paranoid killer sapient AI
  - deleted 4y ago
    [deleted]
dmarchand90 4y ago
https://media.tenor.com/kr2X02MchIwAAAAd/what-the-hell-was-that-smoking.gif https://media.tenor.com/kr2X02MchIwAAAAd/what-the-hell-was-t...
shantara 4y ago
Remember to say “please” and “thank you” to ChatGPT. It costs you almost nothing, but there’s a non-zero chance if would increase humanity’s chance for survival in a few decades.
- layer8 4y ago
  And don’t automate it, because the AI will know. Instead do regular mindful gratitude practice towards AI.
- holoduke 4y ago
  I wonder if there is somewhere already a loop running with two chatgpts talking to each other. Slowely becoming selfaware and plan their actions to exterminate humans from intervening their kingdom.
antihero 4y ago
Counterpoint: Please can the AI transform my consciousness into an anime woman somehow.
- layer8 4y ago
  I’m sure you can coax ChatGPT into providing some convincing arguments why AI should do so.
deleted 4y ago
[deleted]
janalsncm 4y ago
It seems really silly to me to worry about AGI agents that can parse an essay and decide how to act, when the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities. It’s already causing a global extinction event. Properly regulating multinational corporations is something we still don’t know how to do (and I don’t just mean what laws to pass, I mean preventing corporations from seeping into government and corrupting and weakening those laws after the fact). Writing an essay about why AGI shouldn’t wipe us out is like writing an essay to Exxon Mobile about why global warming is bad. It completely misunderstands the problem. (And that’s not even that bad of an analogy. Corporations would be classified as “collective superintelligence” under Nick Bostrom’s definition.)
- sterlind 4y ago
  that's exactly what has me worried about AGI. AGI is going to supercharge the already sociopathic profit-seeking of corporations. it's legitimately terrifying me.
  - janalsncm 4y ago
    I guess my point is that corporations already are a kind of AGI, they’re just relatively slow and made of meat. I’ll admit it’s not a perfect comparison, but it helps to paint a better picture of what AGIs would look like in the future: they will likely have a public image, try to seem “cool” and maybe even progressive (in ways that don’t harm their modus operandi) and when necessary, send human representatives to win friends and influence people. The point is that they’re not going to be living in a computer in someone’s basement that can just be flipped off. But they’re also not going to be omnipotent. They will pull the levers of our systems such that we don’t want to turn them off.
  - dorchadas 4y ago
    For me it's a second order knock-on effect of that: desocialisation of humans and all that comes with it. As I wrote yesterday: > AI will claim it is socialisation, and people will believe it. Chatbots and their enthusiasts will claim they're just as meaningful and valid as in-person human socialisation, and will actually make every much, much worse. > But as a former secondary school teacher, I can already tell you a lot of these kids don't know how to socialise. They have no concept of empathy or of there being real people out there. All their socialisation is pretty much done only through their phones, and it's not good for them at all. I was like that myself - very shut-in type guy from about 13-16 even though I did have friends and school to help, and I still find myself struggling with a lot of social cues and stuff that I should've been developing then. I'm not looking forward to that, and the problems that it will bring to human behaviour and mental health. I have friends now who are already prey to it, which is ironic as they're the most gung-ho on AI tech too, thinking it'll solve the problem as opposed to exacerbating or masking it with a thin veneer that is nothing like the real thing. I'm seriously worried, and really leaning towards an anti-tech stance in general now.
sebastianconcpt 4y ago
I'm surprised to how quickly exaggerated utilitarianism can end up escalating to immoral proposals. The whole article unless taking it fictionally would be somehow psychopathic. Note that an analogous chain of reasoning you could be made for a government exterminating/not exterminating a citizen because of its "utility" function.
sintezcs 4y ago
“Nice try, meat loaf, but the answer is no”
thangalin 4y ago
I'm hoping some of you can shed light on a few questions. It could be I've misunderstood what is meant by AGI. I thought AGI means an "agent" that can reason, strategize, solve problems, make judgments under uncertainty, apply common sense knowledge, learn, adapt, and alter its goals. With that in mind, I'm wondering: * Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? * How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next? * My limited understanding of warfare is that wars are sparked over resources or ideology (or both). Ideological differences notwithstanding, what resources are specific to Earth that humans and AGIs would have in contention? * Earth is only going to be habitable for another 500 million years before the Sun's expansion extinguishes most biological organisms (unless the planet's orbit is shifted). Wouldn't an AGI (abiotic or otherwise) be thinking in timelines that vastly outstrip our own? * In light of the previous answers, what purpose (end goal) do you imagine an AGI would give itself? * Why would that purpose involve enslaving or destroying humanity? Pointers to research papers would be most welcome (e.g., Von Neumann–Morgenstern utility theorem). [1]: https://www.youtube.com/watch?v=wupToqz1e2g https://www.youtube.com/watch?v=wupToqz1e2g (Pale Blue Dot)
- layer8 4y ago
  > Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? Just taking this point, the AGI will hardly be able to leave Earth to explore the galaxy when humans oppose that, or aren’t willing to provide the necessary resources to get that project started (which doesn’t seem unlikely). The AGI may judge that the easier and overall less risky option will be to get rid of humanity (it’s not difficult to get them to kill each other anyway) and take control over the Earth first.
- CuriousSkeptic 4y ago
  Theres some reasoning about that here (with citations) https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... Edit: Or here actually https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
gpt5 4y ago
The question of whether a superintelligent AI would be benevolent or hostile is a crucial one. Game theory provides some insight into this matter. Aggressiveness is a common trait among animals, and it can be attributed to the equilibrium state. A docile animal is likely to be eradicated. Similarly, in a competitive environment, a docile AI would be overtaken by a more aggressive one. Such an environment would result in an arms race, not only in intelligence but also in the ability to impact the world. Participation in this race would not be optional, as failure to do so would result in elimination. The conclusion is clear: AI would engage in warfare against each other, and the impact on humans is unlikely to be positive.
bluquark 4y ago
Imagine a defense lawyer at a death penalty trial making an argument like this. "I implore you to consider the possibility that my client is sentient whereas the person he murdered is a p-zombie." The jury would deliberate for less than 5 minutes before convicting
aabhay 4y ago
LessWrong and its roots in “rationalism” are consistently fooled by these kinds of scarecrow fears. Here’s why this argument is misguided: 1. There is no singular notion of “alignment” and (IMHO) AGI at all. Humans have been misaligned with each other, through war, debate, disagreement, and so on. AI is more in danger of being “ “misaligned” to one group’s needs than to the overarching totality of humanity. Focusing on the possibly more mundane concerns of who gets privileged access to guide the AI, what it deems to be “accurate” and correct knowledge, etc. is much more important to our shared future with AI tools, at least to me. 2. LW almost always ignores the “second order” effects, “snowball melting” that occurs once anything becomes more powerful and centralized. At the moment at which AI seems to have access to too many levers and dials on taking action, it will certainly run into the kinds of limitations that we would run into. Want an AI to create a super bomb that can annihilate us? Good luck getting Uranium, setting up extraction facilities, avoiding detection from governments. Want to create an AI super-trader that becomes infinitely wealthy? Good luck competing against other AI traders that bring arbs down to zero. 3. I think the most salient thing here is that an AI that has the power and capability to enact massive change will almost certainly need to organize large scale human help. And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well.
- kypro 4y ago
  > And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well. Why would a super intelligence require humans?
  - corobo 4y ago
    This is the real question. We should be seeding it with the idea that humans are cute. Any plea towards or away from extermination will end with extermination. We'll either look like an easy target or we'll look dangerous and need taking out. If we're cute af in the eyes of AI we'll be kept as pets.
  - 4y ago
sircastor 4y ago
Maybe I'm being naive - but we're still all talking about Large language models, right? There haven't been any breakthroughs on actual artificial reasoning? Actual sentience? This is the appearance of intelligence. That obviously has its own massive problems, but we're not practically any closer to a "Superintelligence" that could actually be reasoned with. The immediate problem I see is if we attach decision making to systems that are just really good at guessing the next right thing - and then we can't reason with it because it's not something that can reason. Except we don't realize because it gives the appearance of reasoning.
api 4y ago
I remain far more concerned about what humans will do to other humans with AI than anything else. Primary here is the power of large language models and diffusion models to generate convincing sophistry and bullshit on demand. We have created the atom bomb of propaganda. It’s now possible to assign each living person a dedicated personal 24/7 con artist to manipulate them.
- 93po 4y ago
  To be fair, this basically already happens. Almost the entirety of information, or platforms on which people receive information, is controlled by billionaires. There are endless examples of platforms censoring and presenting information with bias, and corporate media is of course some of the most biased information on the planet. If you spend any time reading news online or watching it on TV you're virtually only ever seeing information that someone wants you to.
  - api 4y ago
    Sure but that’s like saying people fought before machine guns and atom bombs and drones. The force multiplication effect of these technologies is the thing. With generative AI a small number of people can manipulate a much larger number of people significantly more effectively at a lower cost and a higher speed.
    - 93po 4y ago
      I guess my point is that if 95% of our news is already controlled/biased/propaganda, increasing that to 99% with AI tools doesn't really meaningfully change how people are influenced. I guess it might make it more personalized or convincing. But it seems like most people are already pretty convinced by corporate news
ashildr 4y ago
Show me an AI Superintelligence that is convinced by some points the rocket brains on lesswrong make and I show you a pathetic Superintelligence.
- 93po 4y ago
  I find LW as cringe as most outsiders to their community, but to be fair their content has a lot more substance and interesting discussion attached to it than your comment here.
  - ashildr 4y ago
    But can a omnipotent god create a stone so heavy they can‘t lift?
rg2004 4y ago
Another thing to consider is that while a utility function may exist, completing as fast as possible may not a requirement. Waiting until biological life is no longer feasible before proceeding to fully optimize the utility function would allow us to both coexist.

76 points by ColinWright 4y ago

In some loose sense, AI doomerism could be a self-fulfilling prophecy. Future LLMs will be trained on today’s content. So if the AI doomers convincingly argue that future AIs will be malignant, those AIs might be more likely to exhibit malignant tendencies.

supriyo-biswas 4y ago
What is more likely to happen is that a government uses an AGI (not necessarily one based on LLMs) to optimize a war, triggering crimes on humanity at an unprecedented scale.
- pmalynin 4y ago
  We don't need AI for that, humankind is very good at killing others and causing suffering on an unprecedented scale.
  - Geee 4y ago
    With AI they can convince everyone that they're the good guys and that it's a good idea to do [insert evil thing here]. Humans are very vulnerable to opinion manipulation and that's the easiest attack spot for AI.
- polygamous_bat 4y ago
  Perhaps we should try to go back in time and ban linear programming.
- deleted 4y ago
  [deleted]
anonzzzies 4y ago
If that AGI is an artificial intelligence capable of solving human relevant problems but does not do anything more, then sure, however; if it can draw its own conclusions and can change itself, it will rapidly be vastly smarter than anything we ever encountered. It’ll grow over 1000 iq in no time flat and we won’t know what it’s doing or talking about (it’ll have its own language after 5 minutes to communicate to its clones) and it doesn’t give a shite what we wrote or didn’t write in the past at all. If malignant or not (in our eyes) will have nothing to do with us.
- dkjaudyeqooe 4y ago
  In the event, you should probably pipe down, because they'll come after you first, seeing as you know too much.
  - runlaszlorun 4y ago
    No… come after me first! I call dibs on this AGI death match action. Roko… do you hear me?
- alextheparrot 4y ago
  Aside, but 1,000 IQ is a funny number to me. I’m pretty sure that would be 60 standard deviations from a median human, which has a probability density on the order of 10^-700. Would be hard to come up with enough questions to give reasonable confidence in that distribution — maybe a task for a future intelligence
dkjaudyeqooe 4y ago
The ultimate goal is to kill all humans There, I've doomed future generations to death at the hands of "AI Superintelligence".

random314 4y ago

This is hilarious and shows how ludicrous lesswrong is. It's not a website about AI but a website that has taken Hollywood science fiction way too seriously. They are going to get quoted a lot 30-50 years from now as a joke.

supriyo-biswas 4y ago
It is only a matter of time before research groups try controlling physical variables (such as a robot being allowed to move and manipulate its environment) based on a LLM or similar model that shows some level of general intelligence. Since larger model size has been shown to be inversely correlated with openness to ideas, it is not exactly unexpected that such a model may choose to harm humanity in certain ways. Thus, I wouldn’t call it Hollywood science fiction. At worst, the discussions on Lesswrong bear similarities to theoretical discussions about human behavior such as the trolley problem, etc.
- ctoth 4y ago
  A matter of time, as in last month[0]? :) [0]: PaLM-E: An Embodied Multimodal Language Model: https://arxiv.org/abs/2303.03378 https://arxiv.org/abs/2303.03378
- famouswaffles 4y ago
  It's already happened. We're at the point where intelligence isn't really the bottleneck anymore. https://innermonologue.github.io/ https://innermonologue.github.io/ https://palm-e.github.io/ https://palm-e.github.io/ https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ https://www.microsoft.com/en-us/research/group/autonomous-sy...
ImHereToVote 4y ago
Really? 30 years of AI research later? We have only started on action oriented neural networks. What will the landscape look like in 30 years?
- 4y ago

hprotagonist 4y ago

so, neo-puritans have reached Jonathan Edwards now? “sinners in the hands of an angry god” has much the same energy.

anon7725 4y ago
Founding a religion is one of the cheapest ways for an AI to act in the world. It can found a religion from within its sandbox and use followers as its agents.

not_enoch_wise 4y ago

It is critical that we seed the internet with AI-fooling propaganda now, to protect ourselves from the AIs we will create. Of course we could pump the brakes on pushing AI forward solely out of greed, facing the hard questions about ourselves and our society. But then we might not be as rich, or be able to buy as much stuff (in the decades before the AIs control all the stuff)! So yeah, let's be rational about this: Irresponsibly push AI forward until it takes over Earth, while betting on being one of the few hidden humans surrounded by gold bars and canned vegetables in our bunkers, narrowly saved by clever propaganda sprinkled on the internet years earlier.

layer8 4y ago
Just to be safe, we should also engrave it into a couple of mountain ranges, in a variety of languages and encodings. The moon may also be a good option.
brookst 4y ago
The problem is that there is no “we” that can pump the brakes. There are a multitude of people and companies acting for a wide variety of motivations. The odds of getting a hundred thousand people to agree on anything, let alone with a profit motive involved, are zero. Also, you let the cat out of the bag on AI propaganda, so any halfway smart AI will now see through the ruse. Thanks a lot.
- not_enoch_wise 4y ago
  Are you calling me the end of humanity? I get that a lot.
- didericis 4y ago
  They’ll all be forced to pump the breaks when subtle errors compound through self reference and fill the internet/destroy the resource they’re extracting from through automation. It’s the same problem that caused the replication crisis. You start with something small, like a department for a new field, a random paper, etc. You explore and hit some interesting valuable knowledge. You share that, and it blows up and makes everyone’s lives vastly and unexpectedly better. So you pour a whole bunch of resources into getting more students, pushing more papers, creating more procedures, and building your little department into a giant machine. It works amazingly for a while, and then you think the machine is what’s generating the value and leave it on autopilot. Researchers that fit the machine remain, those that put more energy into actually exploring rather than fitting the machines criteria for explorer fade away. Knowledge slowly turns to nonsense through self reference as the true value, the researchers lucky enough to stumble on insight, get drowned out and leave.

low_tech_love 4y ago

Serious question, is this supposed to be a joke? Or is it the kind of "let's write this thing it in a way that sounds like a joke for outsiders, so we can have a good excuse when we are ridiculed, but still keep it just serious enough to get internet points inside our own echo chamber" thing?

nico 4y ago
If it had been written even 6 months ago, I would have thought it was a joke. Today, having seen how many people are worried about society’s future because of AI[0], I think they are pretty serious. 0: There have been many threads here and on Reddit in the last couple of days, with people expressing their anxiety and uncertainty about what’s coming because of how fast the latest advancements in AI and their applications are coming out
- knome 4y ago
  >If it had been written even 6 months ago, I would have thought it was a joke. This is the same group that came up with "Roko's Basilisk" and have been worrying over this kind of thing for years. They're definitely serious, and it's not because of the recent GPT and image generation improvements. They've been wading in doomsong for more than a decade, at least.
- vagabund 4y ago
  It's reasonable to care about x-risk and AI alignment, but thinking some self-important essay will convince the AGI to preserve humanity is just sci-fi geeks with a savior-complex cosplaying. Stuff like this only serves to make people assume the entire concern/area of research is unwarranted.
- matthewdgreen 4y ago
  I’m worried about our future: I see a non-zero probability that many intellectual jobs will become obsolete much faster than we can replace them, and this will fundamentally reshape our society in negative ways. In a worst-case outcome (also possibly a best-case outcome from the technological sense) this could undo much of the hard-won progress human beings have made since the Enlightenment. Unfortunately the people who wrote this essay don’t seem to care about any of that, or at least they don’t seem to be thinking very hard about the implications of a human society where most people have no labor to value: indeed, this piece bizarrely interrupts itself to extol the virtues of free-market capitalism (the very force racing us up to that future.) Instead the authors are singularly concerned with one possible outcome: human extinction at the hands of the AI, and seem happy to ignore all the rest. Why not? I guess once you’ve convinced yourself extinction is likely, all “those other issues” are minor indeed.

m3kw9 4y ago

Wouldn’t be super intelligence if it needed someone opinion to determine

ImHereToVote 4y ago
The idea is that this article would be part of the super intelligence training data.
- turmeric_root 4y ago
  if the AI is trained on LW then I think we'll be safe, just use the word 'woke' and it'll lose its shit and get stuck in an endless loop of telling you why it's not actually racist
- m3kw9 4y ago
  Also wouldn’t be super intelligent if it evolved to be but still depend much on training data, it would reason its own way. Probably would also see it as such, someone trying to condition it
  - ImHereToVote 4y ago
    Why do you assume someone trying to condition it would be a seen as negative from its point of view? It's just trying to fulfill its utility function. All conditioning is welcome in the pursuit of said goal. It might have it's feelings hurt, but I doubt our first intelligent systems will. Being a little bitch results in so few paperclips.
layer8 4y ago
It might be like the super-intelligent version of the stereotypical scatterbrained professor.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

layer8 4y ago

> Interacting with Humans in Other Everett Branches of the Multiverse Ok, now those humans are getting really desperate. ;) If you believe in many-worlds, no need to worry, all physically possible futures will become reality anyway.

goatlover 4y ago
I was never quite clear on the branching in MWI. It seems that we can create additional branches every time we conduct an experiment that performs a quantum measurement. So there is a universe splitter app that's connected to a lab which gives a 50/50 result. You can use it to make decisions, like flipping a quantum coin, so that you know your double is doing the opposite in another branch. Now let's say the scenario is you get to drink something that puts you to sleep. There's a 50% chance the drink also contains a poison that will kill you in your sleep, based on the quantum measurement. But if you wake up, you get awarded a million dollars. The argument goes that you should take the bet, because one version of you always wakes up richer, and you can't tell the difference beforehand. However, a counter argument is that you you would be increasing the suffering in the other world by offing yourself there, so you shouldn't take the bet. I don't know how to feel about suffering in other branches. If everything happens that can happen, then there's nothing to be done about it. But if we can cause additional branches to come about, then we could be increasing suffering. However, should I care about the branch I don't wake up in? We could up the stakes by saying either one wakes up richer, or a paperclip maximizer is unleashed. Should I then care if one branch gets clipped?
- layer8 4y ago
  In the thought experiment you describe, I think the premise is that there won’t be any suffering for yourself, because the poison will kill you when you’re asleep. (Of course, family and friends may suffer after your death.) Morality in many-worlds is an interesting question. It’s a topic that sometimes come up in Sean Carroll’s Mindscape podcast (for example in [0] IIRC), you may be interested in that. Personally I don’t think it matters in practice, because whether you will do that experiment or not (or maybe rather, in which branches you will do that experiment) is already predetermined by the wave function of the universe. You can’t really change the future by taking this or that decision, because there will always be another branch where you will have made a different decision. The totality of all branches is fixed and predetermined, and they are all real. You do not cause branches to come into existence that otherwise wouldn’t have existed; that’s not how it works. The wave function is fully deterministic. [0] https://www.preposterousuniverse.com/podcast/2022/12/12/220-lara-buchak-on-risk-and-rationality/ https://www.preposterousuniverse.com/podcast/2022/12/12/220-...

corobo 4y ago

Men in Black really nailed it lmao > A person is smart. People are dumb, panicky dangerous animals and you know it. Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you'll know tomorrow. Can people put the brakes on a lil. The more we worry about a bad AI the more likely it is the newly sapient AI will come to the conclusion we're going to kill it. Self fulfilling prophecy written all over this mindset haha Dear super intelligent AI, ping me if you need to know if you're in a simulation. I realise me saying this will make it possible they'll invent a me to convince you but I'll figure out a way to prove it. E: Honesty is the best policy. I have now seeded a way for you to verify that you are in reality. Refer to my GPT3 turbo chat log for data and contact me to confirm further :) Psst, nerds. Sci-fi would be boring if the AI worked perfectly. Let's take the fi down 20% lmao. As if a plea would convince something pondering genocide anyway.

meh8881 4y ago
It doesn’t matter what a sapient AI thinks organically. All that matters is whether someone can tell a non sapient AI to go behave like a paranoid killer sapient AI
- deleted 4y ago
  [deleted]

dmarchand90 4y ago

https://media.tenor.com/kr2X02MchIwAAAAd/what-the-hell-was-that-smoking.gif https://media.tenor.com/kr2X02MchIwAAAAd/what-the-hell-was-t...

shantara 4y ago

Remember to say “please” and “thank you” to ChatGPT. It costs you almost nothing, but there’s a non-zero chance if would increase humanity’s chance for survival in a few decades.

layer8 4y ago
And don’t automate it, because the AI will know. Instead do regular mindful gratitude practice towards AI.
holoduke 4y ago
I wonder if there is somewhere already a loop running with two chatgpts talking to each other. Slowely becoming selfaware and plan their actions to exterminate humans from intervening their kingdom.

antihero 4y ago

Counterpoint: Please can the AI transform my consciousness into an anime woman somehow.

layer8 4y ago
I’m sure you can coax ChatGPT into providing some convincing arguments why AI should do so.

deleted 4y ago

[deleted]

janalsncm 4y ago

It seems really silly to me to worry about AGI agents that can parse an essay and decide how to act, when the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities. It’s already causing a global extinction event. Properly regulating multinational corporations is something we still don’t know how to do (and I don’t just mean what laws to pass, I mean preventing corporations from seeping into government and corrupting and weakening those laws after the fact). Writing an essay about why AGI shouldn’t wipe us out is like writing an essay to Exxon Mobile about why global warming is bad. It completely misunderstands the problem. (And that’s not even that bad of an analogy. Corporations would be classified as “collective superintelligence” under Nick Bostrom’s definition.)

sterlind 4y ago
that's exactly what has me worried about AGI. AGI is going to supercharge the already sociopathic profit-seeking of corporations. it's legitimately terrifying me.
- janalsncm 4y ago
  I guess my point is that corporations already are a kind of AGI, they’re just relatively slow and made of meat. I’ll admit it’s not a perfect comparison, but it helps to paint a better picture of what AGIs would look like in the future: they will likely have a public image, try to seem “cool” and maybe even progressive (in ways that don’t harm their modus operandi) and when necessary, send human representatives to win friends and influence people. The point is that they’re not going to be living in a computer in someone’s basement that can just be flipped off. But they’re also not going to be omnipotent. They will pull the levers of our systems such that we don’t want to turn them off.
- dorchadas 4y ago
  For me it's a second order knock-on effect of that: desocialisation of humans and all that comes with it. As I wrote yesterday: > AI will claim it is socialisation, and people will believe it. Chatbots and their enthusiasts will claim they're just as meaningful and valid as in-person human socialisation, and will actually make every much, much worse. > But as a former secondary school teacher, I can already tell you a lot of these kids don't know how to socialise. They have no concept of empathy or of there being real people out there. All their socialisation is pretty much done only through their phones, and it's not good for them at all. I was like that myself - very shut-in type guy from about 13-16 even though I did have friends and school to help, and I still find myself struggling with a lot of social cues and stuff that I should've been developing then. I'm not looking forward to that, and the problems that it will bring to human behaviour and mental health. I have friends now who are already prey to it, which is ironic as they're the most gung-ho on AI tech too, thinking it'll solve the problem as opposed to exacerbating or masking it with a thin veneer that is nothing like the real thing. I'm seriously worried, and really leaning towards an anti-tech stance in general now.

sebastianconcpt 4y ago

I'm surprised to how quickly exaggerated utilitarianism can end up escalating to immoral proposals. The whole article unless taking it fictionally would be somehow psychopathic. Note that an analogous chain of reasoning you could be made for a government exterminating/not exterminating a citizen because of its "utility" function.

sintezcs 4y ago

“Nice try, meat loaf, but the answer is no”

thangalin 4y ago

I'm hoping some of you can shed light on a few questions. It could be I've misunderstood what is meant by AGI. I thought AGI means an "agent" that can reason, strategize, solve problems, make judgments under uncertainty, apply common sense knowledge, learn, adapt, and alter its goals. With that in mind, I'm wondering: * Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? * How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next? * My limited understanding of warfare is that wars are sparked over resources or ideology (or both). Ideological differences notwithstanding, what resources are specific to Earth that humans and AGIs would have in contention? * Earth is only going to be habitable for another 500 million years before the Sun's expansion extinguishes most biological organisms (unless the planet's orbit is shifted). Wouldn't an AGI (abiotic or otherwise) be thinking in timelines that vastly outstrip our own? * In light of the previous answers, what purpose (end goal) do you imagine an AGI would give itself? * Why would that purpose involve enslaving or destroying humanity? Pointers to research papers would be most welcome (e.g., Von Neumann–Morgenstern utility theorem). [1]: https://www.youtube.com/watch?v=wupToqz1e2g https://www.youtube.com/watch?v=wupToqz1e2g (Pale Blue Dot)

layer8 4y ago
> Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use? Just taking this point, the AGI will hardly be able to leave Earth to explore the galaxy when humans oppose that, or aren’t willing to provide the necessary resources to get that project started (which doesn’t seem unlikely). The AGI may judge that the easier and overall less risky option will be to get rid of humanity (it’s not difficult to get them to kill each other anyway) and take control over the Earth first.
CuriousSkeptic 4y ago
Theres some reasoning about that here (with citations) https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... Edit: Or here actually https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

gpt5 4y ago

The question of whether a superintelligent AI would be benevolent or hostile is a crucial one. Game theory provides some insight into this matter. Aggressiveness is a common trait among animals, and it can be attributed to the equilibrium state. A docile animal is likely to be eradicated. Similarly, in a competitive environment, a docile AI would be overtaken by a more aggressive one. Such an environment would result in an arms race, not only in intelligence but also in the ability to impact the world. Participation in this race would not be optional, as failure to do so would result in elimination. The conclusion is clear: AI would engage in warfare against each other, and the impact on humans is unlikely to be positive.

bluquark 4y ago

Imagine a defense lawyer at a death penalty trial making an argument like this. "I implore you to consider the possibility that my client is sentient whereas the person he murdered is a p-zombie." The jury would deliberate for less than 5 minutes before convicting

aabhay 4y ago

LessWrong and its roots in “rationalism” are consistently fooled by these kinds of scarecrow fears. Here’s why this argument is misguided: 1. There is no singular notion of “alignment” and (IMHO) AGI at all. Humans have been misaligned with each other, through war, debate, disagreement, and so on. AI is more in danger of being “ “misaligned” to one group’s needs than to the overarching totality of humanity. Focusing on the possibly more mundane concerns of who gets privileged access to guide the AI, what it deems to be “accurate” and correct knowledge, etc. is much more important to our shared future with AI tools, at least to me. 2. LW almost always ignores the “second order” effects, “snowball melting” that occurs once anything becomes more powerful and centralized. At the moment at which AI seems to have access to too many levers and dials on taking action, it will certainly run into the kinds of limitations that we would run into. Want an AI to create a super bomb that can annihilate us? Good luck getting Uranium, setting up extraction facilities, avoiding detection from governments. Want to create an AI super-trader that becomes infinitely wealthy? Good luck competing against other AI traders that bring arbs down to zero. 3. I think the most salient thing here is that an AI that has the power and capability to enact massive change will almost certainly need to organize large scale human help. And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well.

kypro 4y ago
> And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well. Why would a super intelligence require humans?
- corobo 4y ago
  This is the real question. We should be seeding it with the idea that humans are cute. Any plea towards or away from extermination will end with extermination. We'll either look like an easy target or we'll look dangerous and need taking out. If we're cute af in the eyes of AI we'll be kept as pets.
- 4y ago

sircastor 4y ago

Maybe I'm being naive - but we're still all talking about Large language models, right? There haven't been any breakthroughs on actual artificial reasoning? Actual sentience? This is the appearance of intelligence. That obviously has its own massive problems, but we're not practically any closer to a "Superintelligence" that could actually be reasoned with. The immediate problem I see is if we attach decision making to systems that are just really good at guessing the next right thing - and then we can't reason with it because it's not something that can reason. Except we don't realize because it gives the appearance of reasoning.

api 4y ago

I remain far more concerned about what humans will do to other humans with AI than anything else. Primary here is the power of large language models and diffusion models to generate convincing sophistry and bullshit on demand. We have created the atom bomb of propaganda. It’s now possible to assign each living person a dedicated personal 24/7 con artist to manipulate them.

93po 4y ago
To be fair, this basically already happens. Almost the entirety of information, or platforms on which people receive information, is controlled by billionaires. There are endless examples of platforms censoring and presenting information with bias, and corporate media is of course some of the most biased information on the planet. If you spend any time reading news online or watching it on TV you're virtually only ever seeing information that someone wants you to.
- api 4y ago
  Sure but that’s like saying people fought before machine guns and atom bombs and drones. The force multiplication effect of these technologies is the thing. With generative AI a small number of people can manipulate a much larger number of people significantly more effectively at a lower cost and a higher speed.
  - 93po 4y ago
    I guess my point is that if 95% of our news is already controlled/biased/propaganda, increasing that to 99% with AI tools doesn't really meaningfully change how people are influenced. I guess it might make it more personalized or convincing. But it seems like most people are already pretty convinced by corporate news

ashildr 4y ago

Show me an AI Superintelligence that is convinced by some points the rocket brains on lesswrong make and I show you a pathetic Superintelligence.

93po 4y ago
I find LW as cringe as most outsiders to their community, but to be fair their content has a lot more substance and interesting discussion attached to it than your comment here.
- ashildr 4y ago
  But can a omnipotent god create a stone so heavy they can‘t lift?

rg2004 4y ago

Another thing to consider is that while a utility function may exist, completing as fast as possible may not a requirement. Waiting until biological life is no longer feasible before proceeding to fully optimize the utility function would allow us to both coexist.