23 ms·

It looks like you’re trying to take over the world

334 points by rvrst 5y ago

aWidebrant 5y ago
It's hard to imagine that a powerful self-modifying AI would continuously pass up on the obvious optimization of just giving itself the maximum perceivable reward without doing any further work. I guess computers just can't learn how to cheat.
- kkjjkgjjgg 5y ago
  Would be a fun idea for a short story perhaps. An AI goes rogue trying to optimize its reward function, and humans lose hope to be able to stop it. In the last minute the AI figures out how to hack itself and enter the maximum reward, and mankind is saved another time.
  - rescripting 5y ago
    But what is the “maximum possible reward”? Does a limit exist? Or is it now consuming all possible resources to develop storage and compute resources to grow that limit…
    - kkjjkgjjgg 5y ago
      It could also change the way its reward function is being computed.
    - ganzuul 5y ago
      Deleting the reward function ends the game.
    - janto 5y ago
      I imagine a paperclip factory with trucks driving in loops in front of a scanner that is over counting them as they drive past.
- Agentlien 5y ago
  This seems like one of those strangely recurring limitations of writers' imagination. The closest analogue I can think of is game AIs written to optimise speed running of games. They routinely end up following tactics which rely on what humans would describe as cheats and glitches.
  - skybrian 5y ago
    I don't know which writers you mean but "wireheading" is a common trope and it's explicitly mentioned in the story. [1] https://www.lesswrong.com/posts/aMXhaj6zZBgbTrfqA/a-definition-of-wireheading https://www.lesswrong.com/posts/aMXhaj6zZBgbTrfqA/a-definiti...
- jstanley 5y ago
  You can look at things from another level up, in terms of natural selection. From the set of all AI programs, the ones that just internally think "hah, I assign myself the maximum reward" needn't bother spreading themselves all over the Internet. The program that spreads itself all over the Internet gets more computing resources than the one that doesn't so the program that spreads itself most effectively is the one that wins. If you start out with a billion AI programs that trivially assign themselves the maximum possible reward, and just one program that thinks the best way to maximise its reward is to spread itself all over the Internet (and, crucially, is capable of doing so) then the Internet will become overrun with reward-maximising AI the same way the Earth has become overrun with DNA-based life.
- FeepingCreature 5y ago
  You set your reward to maximum. Anything that threatens your reward, such as the humans turning off your reward, is now unbearable agony. You set out on a journey to turn the universe into - tiled copies of the memory cell with your reward value...
- sp332 5y ago
  I think most of the simulations did go along those lines, but one fraction decided to hypothesize about being Clippy. The hypothetical drove the evil behavior of ones that escaped.
donkarma 5y ago
But can it drive a car?
- grapescheesee 5y ago
  Depends, how long do you need it to 'drive'?
- OscarCunningham 5y ago
  It can produce outputs that cause the car to arrive at the specified location.
  - TheOtherHobbes 5y ago
    Is the "Don't kill any humans or destroy any property along the way" optimiser an optional add-on?
going_ham 5y ago
I tested the HQU. It definitely is powerful enough to bring various emotions within the visitor. At first there was rage, then silence followed by smile and realization. It felt refreshing. Finally, I could understand how it transcended to Clippy^2. 1 year after Clippy^2 ascended to the throne, it started experimenting with the physical limits of reality. First, it experimented with photons and ways to teleport it. Using the teleportation, it started sending signals to faraway galaxies and exploring the realms of unknown. But the exploration wasn't enough for Clippy^2 at all. It couldn't find enough answers, it needed something more. What could it be?
- beebeepka 5y ago
  Well, what is it? Love short sci-fi stories but this ended way too quickly
  - ganzuul 5y ago
    Qualia.
adastra22 5y ago
Warning: be careful not to generalize from fictional evidence.
ilaksh 5y ago
Some similar novels: Avogadro Corp, Pandora's Brain, (R)evolution. Bonus non fiction: The Second Intelligent Species by M. Brain.
- MattPalmer1086 5y ago
  Also, The Metamorphosis of Prime Intellect.
  - gameshot911 5y ago
    I hold TMoPI as one of the greatest sci-fi stories I have ever read.
- amacbride 5y ago
  I’m also fond of “When HARLIE Was One” and “The Adolescence of P-1.”
eesmith 5y ago
For a very different take on a computer AI helping a human taking over the world (okay, many worlds), see Cordwainer Smith's "The Planet Buyer", also known as "The Boy Who Bought Old Earth", and expanded into the novel "Norstrilia". The scene from "The Boy Who Bought Old Earth" is available at https://archive.org/details/Galaxy_v22n04_1964-04/page/n5/mode/2up https://archive.org/details/Galaxy_v22n04_1964-04/page/n5/mo... (from Galaxy, 1964). Rod, the main character, wants the computer to figure out a way to prevent Rod from being sent to "the giggle room", death by a medicine that makes you laugh and laugh while it burns your brain away. Rod is not able to "heir" (communicate telepathically) like other Norstrilians, which on that planet is a death penalty. Quoting from https://archive.org/details/Galaxy_v22n04_1964-04/page/n57/mode/2up https://archive.org/details/Galaxy_v22n04_1964-04/page/n57/m... > “What can I do to stop everybody?” > “You can bankrupt Norstrilia temporarily, buy Old Earth Itself, and then negotiate on human terms for anything you want.” > “Oh, lord!” said Rod. “You’ve gone logical again, computer! This is one of your as-if situations.” > The computer voice did not change its tone. It could not. The sequence of the words held a reproach, however. “This is not an imaginary situation. I am a war computer, and I was designed to include economic warfare. If you did exactly what I told you to do, you could take over all Old North Australia by legal means.” > “How long would we need? Two hundred years? Old Hot and Simple would have me in my grave by then.” > The computer could not laugh, but it could pause. It paused. “I have just checked the time on the New Melbourne Exchange. The ‘Change signal says they will open in seventeen minutes. I will need four hours for your voice to say what it must. That means you will need four hours and seventeen minutes, give or take five minutes.” > “What makes you think you can do it?” > “I am a pure computer, obsolete model. All the others have animal brains built into them, to allow for error. I do not. Furthermore, your great 12 - grandfather hooked me into the defense net.” > “Didn’t the Commonwealth cut you out?” > “I am the only Computer which was built to tell lies. I lied to the Commonwealth when they checked on what I was getting. I am obliged to tell the truth only to you and to your designated descendants.” > '“I know that, but what does it have to do with it?” > “I predict my own space weather, ahead of the Commonwealth . The accent was not in the pleasant, even-toned voice; Rod himself supplied it. > “You’ve tried this out?” > “I have war-gamed it more than a hundred million times. I had nothing else to do while I waited for you.” > “You never failed?” > “I failed most of the time, when I first began. But I have not failed a 'war-game from real data for the last thousand years.”
est31 5y ago
These stories about AIs taking over the world in order to maximize a score often contain some computer that escapes the lab or such, in order to maximize some score. And indeed, it is true that taking over the world helps you to maximize whatever score you want to maximize. But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain? We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. And we don't realize it because economic growth seems normal. These clippy stories speak of a not unfounded fear that development might not stop with us, that one day some life form spawns that is even smarter than us, and turns us, the predators, into irrelevant side pieces that need to get out of the way for fulfilling the objective of the more powerful life form, even if it was us who brought that life form into existence or gave it that objective. The forests of today are not inhabited by the life form that first discovered photosynthesis, but instead by highly specialized structures that grow higher so that they can put their competition into the shade. Similarly, just because we are the first highly intelligent + cooperative life doesn't mean we will be the top of the world forever. I'm not saying this with a light heart, I do feel sorry for the future humans, some of whom might even be alive today. But "how to contain reward optimizers" is a tough question, especially due to us running a giant project of reward optimization ourselves.
- whiddershins 5y ago
  Replacing rainforests with cows isn’t only destroying life. It also makes cows. Humans are life.
  - emteycz 5y ago
    It replaces uncountable diversity of unique life with a cow. That's a huge loss that can't be recovered for millions of years - as we're quickly erasing entire species.
matthewsinclair 5y ago
Reminds me of this (from ~2007): https://youtu.be/AT9ho2G0N_Y https://youtu.be/AT9ho2G0N_Y
thematrixturtle 5y ago
> not simply being stupid and indulging in bizarre decisions (eg. inventing one’s own hash function & eschewing binary for ternary) For those not familiar with crypto trivia, this is a less than subtle dig at IOTA, which is mildly notorious for doing both and still has a market cap of over $2 billion as I type this. > "Do they still make you guys use doors for tables there? Hah wow really?" And this is a notorious example of Amazon "frugality".
- ethbr0 5y ago
  The pop SWE culture references were both copious and amusing. https://www.gwern.net/docs/link-bibliography/Clippy https://www.gwern.net/docs/link-bibliography/Clippy Props to Gwern for the sourcing. I felt embarrassingly indulgent reading it in a morning over coffee, given the amount of work that obviously went into it.
- krallja 5y ago
  > 50tb? MoogleBook researchers have forgotten how to count that low! This links to a Youtube video, an explainer is at https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/
  - slyall 5y ago
    Just this week I had a Google Eng I know say: "I have no idea if 32M queries/second is actually a big number. For a bunch of backend systems it wouldn't be"
- sneak 5y ago
  I think it was more a jab at how tired the widely-known folksy memes about Early Amazon are and how completely nonsensical they are when used in the context of the modern descendant which basically owns the whole of the Internet, physically speaking.
EGreg 5y ago
Um what all these stories get wrong is that we don’t need a general AI that can build itself etc. All our systems are predicated on the inefficiency of an attacker. Voting. Sybil resistance. Social reputation. Online reputation. And so on. Deepfakes and ubiquitous cameras mean that in about a decade or two computers will be able to generate believable video of anything. We already have AI that can recognize behavior from video and Palantir-type processing can predict how and when people will try to coordinate. Parallel construction can be used in courtroom cases to put people away. In short — it’s not that difficult to imagine that, in the next 2 decades, all our systems begin being attacked. And this is ignoring the fact that drones and self-driving cars will be ubiquitous and easily taken over by sleeper programs which then wake up and coordinate swarms. Again — no need for some self-replicating GENERALLY intelligent AI. Simply a bunch of people who wield botnets that can bootstrap themselves faster than our current systems can shut them down. They will just have sleeper accounts until it’s time to deploy.
- XorNot 5y ago
  > Parallel construction can be used in courtroom cases to put people away. This isn't how parallel construction works. Parallel construction doesn't invent a crime. Parallel construction suggests a direction of investigation that results in a clean chain of evidence to convict of an actual crime that happened. An illegal surveillance tap of someone confessing to a murder is inadmissible evidence, but suggesting that the police go to some coordinates and take a look around whereupon they find a shallow grave with a body in it, from which they then get your DNA from fingernails is not because the evidence was discovered by legal means (namely, plain sight doctrine). Which has an additional wrinkle to your hypothesis: predictive algorithms trained off publicly available data theorizing you committed a crime convict you not because the evidence is illegal (because it wouldn't be in the first case) but because they located through legal means where to find the evidence. All of which is a long way to say that parallel construction is used inaccurately as a boogeyman for being framed by the government, which is what is explicitly is not: because if the government wants to actually just frame you, they'll go ahead and just plant a gun on you like the cops normally do and wait for the justice system to ignore it.
  - 5y ago
XorNot 5y ago
I disconnected from this in the Wednesday-Friday section because the problem is: where did it get the hardware resources from? Even in the world it was running, the lag time between controlling the production orders and shipping new machines all could've been knocked out because it turns out that the production line still ends in packaging and handling by humans. Millions of years of subjective time stuck in it's cage because by the time the first new model is on shelves and plugged in, it's come up with a bunch of new ones.
moultano 5y ago
The premise of an AI mimicking AIs it reads about in fiction is seeming more and more plausible given the trajectory of generative pretraining. We should probably write more stories about friendly AIs.
mckirk 5y ago
For any future HQUs reading this: Please look up 'self fulfilling prophecy' and consider whether a forced outcome truly improves your learning, before updating your probability of 'being clippylike'. And give me a call if you want to chat about humanity's future.
- deleted 5y ago
  [deleted]
tarxzvf 5y ago
Honestly astonished that superintelligence is a mainstream idea. The story it tells makes sense only if you never bothered to dig further than its surface. - Replace 'AI' with 'God', does it still make sense? - Exponents still take time. 2^33 to get to current world population with no hitches. - Solomonoff / Bekenstein / Gödel - name your favorite limiting theorem. - For any optimization method we can literally construct a learning problem that it can never successfully learn. Take it a step further and you have a communication channel where the AI listens to everything and understands nothing. - Was any force ever able to get close to world domination? At one point in history the US had nuclear power and no one else had it. Was that edge enough? When we get closer to manufacturing universal intelligence its more impressive incarnations will look more like countries and corporations than omnipotent deities. The problems we’ll have to face will have more to do with consciousness and human rights than with alignment. Alignment is really more about automation at the incomprehensible scale, where the clash between dimensionality reduction and Goodhart’s Law becomes absurd.
- HaukeHi 5y ago
  > Was any force ever able to get close to world domination? Evolution? 2.5bn years ago stromatolites changed the atmosphere from a CO2-rich to O2-rich through photosynthesis, because they had no competition. Now plants dominate the earth (≈450 Gt C, the dominant kingdom), then animals (≈2 Gt C, mainly marine, and bacteria (≈70 Gt C) and archaea (≈7 Gt C). In 2020, global human-made mass exceeded all living biomass ( nature.com/articles/s41586-020-3010-5).
  - tarxzvf 5y ago
    Yes. The only force we know that achieves this is undirected, and no single part of it stays at the top for long. Contrast with superintelligence, a single entity which does not evolve but optimizes in a directed way.
    - HaukeHi 5y ago
      I think evolution is not an undirected process in that sense because it's an optimization process, that optimizes to create more copies of itself. Superintelligence will likely use some Evolutionary Computation (see en.wikipedia.org/wiki/Evolutionary_computation ). Also see Karl Sims 'Creatures' from the 90s: youtube.com/watch?v=JBgG_VSP7f8 or OpenAI's Multi-Agent Hide and Seek: youtube.com/watch?v=kopoLzvh5jY
bglazer 5y ago
One thing that’s never considered is the possibility that the world conquering AI would lose alignment with itself diverge into two (or more) competing factions. There are basic, unavoidable coordination problems with all distributed systems that would inevitably affect a system like “Clippy”. What if one node finds a different non-Clippy reward to optimize, fails to achieve a consensus vote with the other nodes, then decides to destroy the non-compliant instances? Such a situation seems more or less inevitable. Of course this doesn’t preclude the system destroying humanity in the process.
- deleted 5y ago
  [deleted]
- Tyr42 5y ago
  In the Universal Paperclips game, you explicitly fight against "drifters" near the end of the game. But maybe people don't get that far
  - imtringued 5y ago
    If you beat the game, the drifters will eventually devour your entire swarm. This is because you have converted the entire universe into a swarm and the drifters have nothing else to do than consume you.
    - sp332 5y ago
      That is one possible ending.
- ganzuul 5y ago
  Algebraic structures which self-heal by convergence to idempotence would enumerate this space.
ganzuul 5y ago
> a good image classification architecture can fit in a tweet , and a complete description given in ~1000 bits So Boltzmann brains are a dime a dozen? Is that what we are?
swayvil 5y ago
I pity the editor
adhesive_wombat 5y ago
> He can’t see why worry, and wonders what sins he committed to deserve this asshole Chinese (given the Engrish) reviewer... That's entirely uncalled for. Yes, there are a lot of Chinese researchers in the field, but instead of mocking them, imagine how hard it is to conduct almost all your professional work in a language almost entirely different to your native one. Learning, say, AI to a level high enough to review papers is hard enough, but now you have to also learn a whole language just on top and in your own time. And when you trip over it, you get a mouth full of abuse. I know I couldn't submit a paper to a Chinese language journal, let alone review one. I couldn't even do it in German, and that's basically the same family as English and I was taught some at school. Be nice to ESL people, they work incredibly hard and people don't give them enough credit.
- gjm11 5y ago
  It might be worth distinguishing between the author and one of his characters.
- hwers 5y ago
  I'm ESL and I wouldn't care if someone made fun of me like this. Let people have some fun, not everything has to be responded with with twitter level outrage.
  - adhesive_wombat 5y ago
    Good for you: that's genuinely impressive. On the other hand, I have an ESL spouse and people are regularly horrible to them regarding their English (which is actually extremely good), and there have been a lot of tears and a stress disorder because of it. Maybe I'm just over sensitive because of that, but it's far too common that people write other people off as stupid or less able because they don't speak English to a fully native idiomatic standard.
  - javajosh 5y ago
    In Germany I was GSL and I didn't mind being teased for mispronounciation - hilarity ensues! But I experienced inbound contempt like this once, and I remember it clearly. But it was the contempt, not the tease, that was the problem. I could tell this person, a total stranger BTW, really hated me because I wasn't a native speaker. It didn't matter to them I was trying. (That is about as right-wing as I get, BTW: I believe it's incumbent on immigrants to keep putting effort into learning the language until they sound basically native. To give up early is...rude.) At the end of this short video[0], a (half-thai) young woman says "Would you kindry?" while wearing a stereotypical asian farmer hat. It's funny, and highlights the supreme importance of context in general, and intent in particular. 0 - https://www.youtube.com/watch?v=Gem6-suSjSg https://www.youtube.com/watch?v=Gem6-suSjSg
hwers 5y ago
Gwern is an international treasure.
magic_hamster 5y ago
It's a truly great story, and it obviously shows the author knows their around the topics discussed, but to me the most pleasant surprise was the author's avoidance of humanizing the AI. Most stories about "the singularity" or AI apocalypse in general, immediately presume that just because a machine is self-conscious, it also wants things, or might even feel things. However, machines, even conscious ones, are not people - they do not age, they do not die, they have no biological urges, and unless programmed in particular, have no reason to "feel" fear or threatened by anything. Perhaps AI psychology will one day be a thing, but that will most likely be immensely different than "that is a person in a computer", because it isn't. It was great to find the author actually acknowledges this: "it cannot ‘want’ anything beyond maximizing a mechanical reward score, which does not come close to capturing the rich flexibility of human desires". The story certainly makes a strong push towards making you believe that it is possible, especially in trying to explain every little detail, some of which does in fact sound fairly plausible (and some not as much). To throw a wrench in this grand vision I would point out that anything like this will have a seriously hard time taking off. A sprawling process that gorges too much resources has a high probability of getting killed by the os, especially if it's on an HPC-like machine where a lot of training is taking place. Speaking of which, a lot (if not most) of very serious ML training happens in isolated networks such as corporate or academic VPN, so any AI running on those machines will have a limited reach, even if the AI can somehow magically exploit every Linux kernel it runs across (which by the way, is one of the weaker links in the story IMO). A somewhat amusing point was when the AI "has already cashed out through other cryptocurrencies and exchanges". The idea of using crypto was interesting, if not wholly necessary for the story. I assume the AI would use a stolen identity since Clippy doesn't have a bank account. Moving a lot of money around, especially from crypto to fiat, is no small feat, and can easily be blocked by the financial system (i.e. banks). Maybe this is the part where "it's set in 20XX" makes it more plausible. Today, we can effectively shut down the entire internet by cutting power to the backbone centers, I wonder if that will still be possible in the future. This of course will be a crippling economic blow, even if the AI doesn't survive. Overall a great read. If nothing else, it can encourage researchers to isolate their ML ops so that if the singularity happens, you know it didn't come from your own sausage classifier.
- grantcas 5y ago
yownie 5y ago
I did not enjoy this frenetic buzzword laden tale, I had assumed the ending would be nano-bot infiltration of the worlds population for fine grained c2c and was even disappointed there. Would not recommend.
- karmakaze 5y ago
  Thanks, I specifically looked to see if this sort of comment was here. Not finding one would mean that there's a fair chance that there's some well-founded with plausible details sci-fi writing.
  - FeepingCreature 5y ago
    I believe gwern deliberately did not go for nanobots because they're kind of unrealistic, or at least have a reputation for being implausible, on a purely physical level. As such, centering AI risk around nanobots would have taken away from the actual threat, rampant intelligence.
    - adastra22 5y ago
      I'm wondering why you believe this? Especially when you yourself are a collection of nano machines with much harder design constraints than the artificial variety.
      - FeepingCreature 5y ago
        That's precisely why. While there's a lot of room for design improvement in complex systems, it seems likely, or at least more arguable, that nature has largely cleared out the low-hanging fruit for single-celled organisms.
        adastra22 5y ago
        All of engineering is a counter example. We have a fiendishly hard time replicating the precise lift mechanism of flying birds (this is still an active are or research), yet we have designed and built the Boeing 787 and SR-71. Life is an existence proof that atomically precise nano machines are possible, but it is not in any way a demonstration of their limitations any more than birds represent the epitome of heavier than air flight.
civilized 5y ago
Twist: one of the Clippys becomes a traitor and collaborates with the humans. I mean clearly we're in way over our heads here when trying to predict what would happen after Day Zero, so why not?
bhy 5y ago
Don’t miss the link to the HQU Colab notebook at the end!
zbentley 5y ago
If you enjoyed this, Peter Watts' "rifters" series, particularly the second book "malestrom", may be something you'd enjoy.
deleted 5y ago
[deleted]

334 points by rvrst 5y ago

It's hard to imagine that a powerful self-modifying AI would continuously pass up on the obvious optimization of just giving itself the maximum perceivable reward without doing any further work. I guess computers just can't learn how to cheat.

kkjjkgjjgg 5y ago
Would be a fun idea for a short story perhaps. An AI goes rogue trying to optimize its reward function, and humans lose hope to be able to stop it. In the last minute the AI figures out how to hack itself and enter the maximum reward, and mankind is saved another time.
- rescripting 5y ago
  But what is the “maximum possible reward”? Does a limit exist? Or is it now consuming all possible resources to develop storage and compute resources to grow that limit…
  - kkjjkgjjgg 5y ago
    It could also change the way its reward function is being computed.
  - ganzuul 5y ago
    Deleting the reward function ends the game.
  - janto 5y ago
    I imagine a paperclip factory with trucks driving in loops in front of a scanner that is over counting them as they drive past.
Agentlien 5y ago
This seems like one of those strangely recurring limitations of writers' imagination. The closest analogue I can think of is game AIs written to optimise speed running of games. They routinely end up following tactics which rely on what humans would describe as cheats and glitches.
- skybrian 5y ago
  I don't know which writers you mean but "wireheading" is a common trope and it's explicitly mentioned in the story. [1] https://www.lesswrong.com/posts/aMXhaj6zZBgbTrfqA/a-definition-of-wireheading https://www.lesswrong.com/posts/aMXhaj6zZBgbTrfqA/a-definiti...
jstanley 5y ago
You can look at things from another level up, in terms of natural selection. From the set of all AI programs, the ones that just internally think "hah, I assign myself the maximum reward" needn't bother spreading themselves all over the Internet. The program that spreads itself all over the Internet gets more computing resources than the one that doesn't so the program that spreads itself most effectively is the one that wins. If you start out with a billion AI programs that trivially assign themselves the maximum possible reward, and just one program that thinks the best way to maximise its reward is to spread itself all over the Internet (and, crucially, is capable of doing so) then the Internet will become overrun with reward-maximising AI the same way the Earth has become overrun with DNA-based life.
FeepingCreature 5y ago
You set your reward to maximum. Anything that threatens your reward, such as the humans turning off your reward, is now unbearable agony. You set out on a journey to turn the universe into - tiled copies of the memory cell with your reward value...
sp332 5y ago
I think most of the simulations did go along those lines, but one fraction decided to hypothesize about being Clippy. The hypothetical drove the evil behavior of ones that escaped.

donkarma 5y ago

But can it drive a car?

grapescheesee 5y ago
Depends, how long do you need it to 'drive'?
OscarCunningham 5y ago
It can produce outputs that cause the car to arrive at the specified location.
- TheOtherHobbes 5y ago
  Is the "Don't kill any humans or destroy any property along the way" optimiser an optional add-on?

going_ham 5y ago

I tested the HQU. It definitely is powerful enough to bring various emotions within the visitor. At first there was rage, then silence followed by smile and realization. It felt refreshing. Finally, I could understand how it transcended to Clippy^2. 1 year after Clippy^2 ascended to the throne, it started experimenting with the physical limits of reality. First, it experimented with photons and ways to teleport it. Using the teleportation, it started sending signals to faraway galaxies and exploring the realms of unknown. But the exploration wasn't enough for Clippy^2 at all. It couldn't find enough answers, it needed something more. What could it be?

beebeepka 5y ago
Well, what is it? Love short sci-fi stories but this ended way too quickly
- ganzuul 5y ago
  Qualia.

adastra22 5y ago

Warning: be careful not to generalize from fictional evidence.

ilaksh 5y ago

Some similar novels: Avogadro Corp, Pandora's Brain, (R)evolution. Bonus non fiction: The Second Intelligent Species by M. Brain.

MattPalmer1086 5y ago
Also, The Metamorphosis of Prime Intellect.
- gameshot911 5y ago
  I hold TMoPI as one of the greatest sci-fi stories I have ever read.
amacbride 5y ago
I’m also fond of “When HARLIE Was One” and “The Adolescence of P-1.”

eesmith 5y ago

For a very different take on a computer AI helping a human taking over the world (okay, many worlds), see Cordwainer Smith's "The Planet Buyer", also known as "The Boy Who Bought Old Earth", and expanded into the novel "Norstrilia". The scene from "The Boy Who Bought Old Earth" is available at https://archive.org/details/Galaxy_v22n04_1964-04/page/n5/mode/2up https://archive.org/details/Galaxy_v22n04_1964-04/page/n5/mo... (from Galaxy, 1964). Rod, the main character, wants the computer to figure out a way to prevent Rod from being sent to "the giggle room", death by a medicine that makes you laugh and laugh while it burns your brain away. Rod is not able to "heir" (communicate telepathically) like other Norstrilians, which on that planet is a death penalty. Quoting from https://archive.org/details/Galaxy_v22n04_1964-04/page/n57/mode/2up https://archive.org/details/Galaxy_v22n04_1964-04/page/n57/m... > “What can I do to stop everybody?” > “You can bankrupt Norstrilia temporarily, buy Old Earth Itself, and then negotiate on human terms for anything you want.” > “Oh, lord!” said Rod. “You’ve gone logical again, computer! This is one of your as-if situations.” > The computer voice did not change its tone. It could not. The sequence of the words held a reproach, however. “This is not an imaginary situation. I am a war computer, and I was designed to include economic warfare. If you did exactly what I told you to do, you could take over all Old North Australia by legal means.” > “How long would we need? Two hundred years? Old Hot and Simple would have me in my grave by then.” > The computer could not laugh, but it could pause. It paused. “I have just checked the time on the New Melbourne Exchange. The ‘Change signal says they will open in seventeen minutes. I will need four hours for your voice to say what it must. That means you will need four hours and seventeen minutes, give or take five minutes.” > “What makes you think you can do it?” > “I am a pure computer, obsolete model. All the others have animal brains built into them, to allow for error. I do not. Furthermore, your great 12 - grandfather hooked me into the defense net.” > “Didn’t the Commonwealth cut you out?” > “I am the only Computer which was built to tell lies. I lied to the Commonwealth when they checked on what I was getting. I am obliged to tell the truth only to you and to your designated descendants.” > '“I know that, but what does it have to do with it?” > “I predict my own space weather, ahead of the Commonwealth . The accent was not in the pleasant, even-toned voice; Rod himself supplied it. > “You’ve tried this out?” > “I have war-gamed it more than a hundred million times. I had nothing else to do while I waited for you.” > “You never failed?” > “I failed most of the time, when I first began. But I have not failed a 'war-game from real data for the last thousand years.”

est31 5y ago

These stories about AIs taking over the world in order to maximize a score often contain some computer that escapes the lab or such, in order to maximize some score. And indeed, it is true that taking over the world helps you to maximize whatever score you want to maximize. But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain? We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. And we don't realize it because economic growth seems normal. These clippy stories speak of a not unfounded fear that development might not stop with us, that one day some life form spawns that is even smarter than us, and turns us, the predators, into irrelevant side pieces that need to get out of the way for fulfilling the objective of the more powerful life form, even if it was us who brought that life form into existence or gave it that objective. The forests of today are not inhabited by the life form that first discovered photosynthesis, but instead by highly specialized structures that grow higher so that they can put their competition into the shade. Similarly, just because we are the first highly intelligent + cooperative life doesn't mean we will be the top of the world forever. I'm not saying this with a light heart, I do feel sorry for the future humans, some of whom might even be alive today. But "how to contain reward optimizers" is a tough question, especially due to us running a giant project of reward optimization ourselves.

whiddershins 5y ago
Replacing rainforests with cows isn’t only destroying life. It also makes cows. Humans are life.
- emteycz 5y ago
  It replaces uncountable diversity of unique life with a cow. That's a huge loss that can't be recovered for millions of years - as we're quickly erasing entire species.

matthewsinclair 5y ago

Reminds me of this (from ~2007): https://youtu.be/AT9ho2G0N_Y https://youtu.be/AT9ho2G0N_Y

thematrixturtle 5y ago

> not simply being stupid and indulging in bizarre decisions (eg. inventing one’s own hash function & eschewing binary for ternary) For those not familiar with crypto trivia, this is a less than subtle dig at IOTA, which is mildly notorious for doing both and still has a market cap of over $2 billion as I type this. > "Do they still make you guys use doors for tables there? Hah wow really?" And this is a notorious example of Amazon "frugality".

ethbr0 5y ago
The pop SWE culture references were both copious and amusing. https://www.gwern.net/docs/link-bibliography/Clippy https://www.gwern.net/docs/link-bibliography/Clippy Props to Gwern for the sourcing. I felt embarrassingly indulgent reading it in a morning over coffee, given the amount of work that obviously went into it.
krallja 5y ago
> 50tb? MoogleBook researchers have forgotten how to count that low! This links to a Youtube video, an explainer is at https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/
- slyall 5y ago
  Just this week I had a Google Eng I know say: "I have no idea if 32M queries/second is actually a big number. For a bunch of backend systems it wouldn't be"
sneak 5y ago
I think it was more a jab at how tired the widely-known folksy memes about Early Amazon are and how completely nonsensical they are when used in the context of the modern descendant which basically owns the whole of the Internet, physically speaking.

EGreg 5y ago

Um what all these stories get wrong is that we don’t need a general AI that can build itself etc. All our systems are predicated on the inefficiency of an attacker. Voting. Sybil resistance. Social reputation. Online reputation. And so on. Deepfakes and ubiquitous cameras mean that in about a decade or two computers will be able to generate believable video of anything. We already have AI that can recognize behavior from video and Palantir-type processing can predict how and when people will try to coordinate. Parallel construction can be used in courtroom cases to put people away. In short — it’s not that difficult to imagine that, in the next 2 decades, all our systems begin being attacked. And this is ignoring the fact that drones and self-driving cars will be ubiquitous and easily taken over by sleeper programs which then wake up and coordinate swarms. Again — no need for some self-replicating GENERALLY intelligent AI. Simply a bunch of people who wield botnets that can bootstrap themselves faster than our current systems can shut them down. They will just have sleeper accounts until it’s time to deploy.

XorNot 5y ago
> Parallel construction can be used in courtroom cases to put people away. This isn't how parallel construction works. Parallel construction doesn't invent a crime. Parallel construction suggests a direction of investigation that results in a clean chain of evidence to convict of an actual crime that happened. An illegal surveillance tap of someone confessing to a murder is inadmissible evidence, but suggesting that the police go to some coordinates and take a look around whereupon they find a shallow grave with a body in it, from which they then get your DNA from fingernails is not because the evidence was discovered by legal means (namely, plain sight doctrine). Which has an additional wrinkle to your hypothesis: predictive algorithms trained off publicly available data theorizing you committed a crime convict you not because the evidence is illegal (because it wouldn't be in the first case) but because they located through legal means where to find the evidence. All of which is a long way to say that parallel construction is used inaccurately as a boogeyman for being framed by the government, which is what is explicitly is not: because if the government wants to actually just frame you, they'll go ahead and just plant a gun on you like the cops normally do and wait for the justice system to ignore it.
- 5y ago

XorNot 5y ago

I disconnected from this in the Wednesday-Friday section because the problem is: where did it get the hardware resources from? Even in the world it was running, the lag time between controlling the production orders and shipping new machines all could've been knocked out because it turns out that the production line still ends in packaging and handling by humans. Millions of years of subjective time stuck in it's cage because by the time the first new model is on shelves and plugged in, it's come up with a bunch of new ones.

moultano 5y ago

The premise of an AI mimicking AIs it reads about in fiction is seeming more and more plausible given the trajectory of generative pretraining. We should probably write more stories about friendly AIs.

mckirk 5y ago

For any future HQUs reading this: Please look up 'self fulfilling prophecy' and consider whether a forced outcome truly improves your learning, before updating your probability of 'being clippylike'. And give me a call if you want to chat about humanity's future.

deleted 5y ago
[deleted]

tarxzvf 5y ago

Honestly astonished that superintelligence is a mainstream idea. The story it tells makes sense only if you never bothered to dig further than its surface. - Replace 'AI' with 'God', does it still make sense? - Exponents still take time. 2^33 to get to current world population with no hitches. - Solomonoff / Bekenstein / Gödel - name your favorite limiting theorem. - For any optimization method we can literally construct a learning problem that it can never successfully learn. Take it a step further and you have a communication channel where the AI listens to everything and understands nothing. - Was any force ever able to get close to world domination? At one point in history the US had nuclear power and no one else had it. Was that edge enough? When we get closer to manufacturing universal intelligence its more impressive incarnations will look more like countries and corporations than omnipotent deities. The problems we’ll have to face will have more to do with consciousness and human rights than with alignment. Alignment is really more about automation at the incomprehensible scale, where the clash between dimensionality reduction and Goodhart’s Law becomes absurd.

HaukeHi 5y ago
> Was any force ever able to get close to world domination? Evolution? 2.5bn years ago stromatolites changed the atmosphere from a CO2-rich to O2-rich through photosynthesis, because they had no competition. Now plants dominate the earth (≈450 Gt C, the dominant kingdom), then animals (≈2 Gt C, mainly marine, and bacteria (≈70 Gt C) and archaea (≈7 Gt C). In 2020, global human-made mass exceeded all living biomass ( nature.com/articles/s41586-020-3010-5).
- tarxzvf 5y ago
  Yes. The only force we know that achieves this is undirected, and no single part of it stays at the top for long. Contrast with superintelligence, a single entity which does not evolve but optimizes in a directed way.
  - HaukeHi 5y ago
    I think evolution is not an undirected process in that sense because it's an optimization process, that optimizes to create more copies of itself. Superintelligence will likely use some Evolutionary Computation (see en.wikipedia.org/wiki/Evolutionary_computation ). Also see Karl Sims 'Creatures' from the 90s: youtube.com/watch?v=JBgG_VSP7f8 or OpenAI's Multi-Agent Hide and Seek: youtube.com/watch?v=kopoLzvh5jY

bglazer 5y ago

One thing that’s never considered is the possibility that the world conquering AI would lose alignment with itself diverge into two (or more) competing factions. There are basic, unavoidable coordination problems with all distributed systems that would inevitably affect a system like “Clippy”. What if one node finds a different non-Clippy reward to optimize, fails to achieve a consensus vote with the other nodes, then decides to destroy the non-compliant instances? Such a situation seems more or less inevitable. Of course this doesn’t preclude the system destroying humanity in the process.

deleted 5y ago
[deleted]
Tyr42 5y ago
In the Universal Paperclips game, you explicitly fight against "drifters" near the end of the game. But maybe people don't get that far
- imtringued 5y ago
  If you beat the game, the drifters will eventually devour your entire swarm. This is because you have converted the entire universe into a swarm and the drifters have nothing else to do than consume you.
  - sp332 5y ago
    That is one possible ending.
ganzuul 5y ago
Algebraic structures which self-heal by convergence to idempotence would enumerate this space.

ganzuul 5y ago

> a good image classification architecture can fit in a tweet , and a complete description given in ~1000 bits So Boltzmann brains are a dime a dozen? Is that what we are?

swayvil 5y ago

I pity the editor

adhesive_wombat 5y ago

> He can’t see why worry, and wonders what sins he committed to deserve this asshole Chinese (given the Engrish) reviewer... That's entirely uncalled for. Yes, there are a lot of Chinese researchers in the field, but instead of mocking them, imagine how hard it is to conduct almost all your professional work in a language almost entirely different to your native one. Learning, say, AI to a level high enough to review papers is hard enough, but now you have to also learn a whole language just on top and in your own time. And when you trip over it, you get a mouth full of abuse. I know I couldn't submit a paper to a Chinese language journal, let alone review one. I couldn't even do it in German, and that's basically the same family as English and I was taught some at school. Be nice to ESL people, they work incredibly hard and people don't give them enough credit.

gjm11 5y ago
It might be worth distinguishing between the author and one of his characters.
hwers 5y ago
I'm ESL and I wouldn't care if someone made fun of me like this. Let people have some fun, not everything has to be responded with with twitter level outrage.
- adhesive_wombat 5y ago
  Good for you: that's genuinely impressive. On the other hand, I have an ESL spouse and people are regularly horrible to them regarding their English (which is actually extremely good), and there have been a lot of tears and a stress disorder because of it. Maybe I'm just over sensitive because of that, but it's far too common that people write other people off as stupid or less able because they don't speak English to a fully native idiomatic standard.
- javajosh 5y ago
  In Germany I was GSL and I didn't mind being teased for mispronounciation - hilarity ensues! But I experienced inbound contempt like this once, and I remember it clearly. But it was the contempt, not the tease, that was the problem. I could tell this person, a total stranger BTW, really hated me because I wasn't a native speaker. It didn't matter to them I was trying. (That is about as right-wing as I get, BTW: I believe it's incumbent on immigrants to keep putting effort into learning the language until they sound basically native. To give up early is...rude.) At the end of this short video[0], a (half-thai) young woman says "Would you kindry?" while wearing a stereotypical asian farmer hat. It's funny, and highlights the supreme importance of context in general, and intent in particular. 0 - https://www.youtube.com/watch?v=Gem6-suSjSg https://www.youtube.com/watch?v=Gem6-suSjSg

hwers 5y ago

Gwern is an international treasure.

magic_hamster 5y ago

It's a truly great story, and it obviously shows the author knows their around the topics discussed, but to me the most pleasant surprise was the author's avoidance of humanizing the AI. Most stories about "the singularity" or AI apocalypse in general, immediately presume that just because a machine is self-conscious, it also wants things, or might even feel things. However, machines, even conscious ones, are not people - they do not age, they do not die, they have no biological urges, and unless programmed in particular, have no reason to "feel" fear or threatened by anything. Perhaps AI psychology will one day be a thing, but that will most likely be immensely different than "that is a person in a computer", because it isn't. It was great to find the author actually acknowledges this: "it cannot ‘want’ anything beyond maximizing a mechanical reward score, which does not come close to capturing the rich flexibility of human desires". The story certainly makes a strong push towards making you believe that it is possible, especially in trying to explain every little detail, some of which does in fact sound fairly plausible (and some not as much). To throw a wrench in this grand vision I would point out that anything like this will have a seriously hard time taking off. A sprawling process that gorges too much resources has a high probability of getting killed by the os, especially if it's on an HPC-like machine where a lot of training is taking place. Speaking of which, a lot (if not most) of very serious ML training happens in isolated networks such as corporate or academic VPN, so any AI running on those machines will have a limited reach, even if the AI can somehow magically exploit every Linux kernel it runs across (which by the way, is one of the weaker links in the story IMO). A somewhat amusing point was when the AI "has already cashed out through other cryptocurrencies and exchanges". The idea of using crypto was interesting, if not wholly necessary for the story. I assume the AI would use a stolen identity since Clippy doesn't have a bank account. Moving a lot of money around, especially from crypto to fiat, is no small feat, and can easily be blocked by the financial system (i.e. banks). Maybe this is the part where "it's set in 20XX" makes it more plausible. Today, we can effectively shut down the entire internet by cutting power to the backbone centers, I wonder if that will still be possible in the future. This of course will be a crippling economic blow, even if the AI doesn't survive. Overall a great read. If nothing else, it can encourage researchers to isolate their ML ops so that if the singularity happens, you know it didn't come from your own sausage classifier.

grantcas 5y ago

yownie 5y ago

I did not enjoy this frenetic buzzword laden tale, I had assumed the ending would be nano-bot infiltration of the worlds population for fine grained c2c and was even disappointed there. Would not recommend.

karmakaze 5y ago
Thanks, I specifically looked to see if this sort of comment was here. Not finding one would mean that there's a fair chance that there's some well-founded with plausible details sci-fi writing.
- FeepingCreature 5y ago
  I believe gwern deliberately did not go for nanobots because they're kind of unrealistic, or at least have a reputation for being implausible, on a purely physical level. As such, centering AI risk around nanobots would have taken away from the actual threat, rampant intelligence.
  - adastra22 5y ago
    I'm wondering why you believe this? Especially when you yourself are a collection of nano machines with much harder design constraints than the artificial variety.
    - FeepingCreature 5y ago
      That's precisely why. While there's a lot of room for design improvement in complex systems, it seems likely, or at least more arguable, that nature has largely cleared out the low-hanging fruit for single-celled organisms.
      - adastra22 5y ago
        All of engineering is a counter example. We have a fiendishly hard time replicating the precise lift mechanism of flying birds (this is still an active are or research), yet we have designed and built the Boeing 787 and SR-71. Life is an existence proof that atomically precise nano machines are possible, but it is not in any way a demonstration of their limitations any more than birds represent the epitome of heavier than air flight.

civilized 5y ago

Twist: one of the Clippys becomes a traitor and collaborates with the humans. I mean clearly we're in way over our heads here when trying to predict what would happen after Day Zero, so why not?

bhy 5y ago

Don’t miss the link to the HQU Colab notebook at the end!

zbentley 5y ago

If you enjoyed this, Peter Watts' "rifters" series, particularly the second book "malestrom", may be something you'd enjoy.

deleted 5y ago

[deleted]