24 ms·

These stories about AIs taking over the world in order to maximize a score often contain some computer that escapes the lab or such, in order to maximize some s

by est31 5y ago

These stories about AIs taking over the world in order to maximize a score often contain some computer that escapes the lab or such, in order to maximize some score. And indeed, it is true that taking over the world helps you to maximize whatever score you want to maximize. But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain? We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. And we don't realize it because economic growth seems normal. These clippy stories speak of a not unfounded fear that development might not stop with us, that one day some life form spawns that is even smarter than us, and turns us, the predators, into irrelevant side pieces that need to get out of the way for fulfilling the objective of the more powerful life form, even if it was us who brought that life form into existence or gave it that objective. The forests of today are not inhabited by the life form that first discovered photosynthesis, but instead by highly specialized structures that grow higher so that they can put their competition into the shade. Similarly, just because we are the first highly intelligent + cooperative life doesn't mean we will be the top of the world forever. I'm not saying this with a light heart, I do feel sorry for the future humans, some of whom might even be alive today. But "how to contain reward optimizers" is a tough question, especially due to us running a giant project of reward optimization ourselves.

whiddershins 5y ago
Replacing rainforests with cows isn’t only destroying life. It also makes cows. Humans are life.
- emteycz 5y ago
  It replaces uncountable diversity of unique life with a cow. That's a huge loss that can't be recovered for millions of years - as we're quickly erasing entire species.
  - bitcoin_anon 5y ago
    Not if you’re hungry.
    - emteycz 5y ago
      We have much better ways to make food than cows. If you're hungry you don't go raise cows. You go farming. Much less space needed for the same result, it's much quicker and with way less risk.
      - bitcoin_anon 5y ago
        Yet, people are still raising them for food. The species under question doesn’t matter. Farming is also destroying diversity so that people can eat. There is nothing wrong here. It is the way of nature. As you read this, bacteria, viruses, and fungi are trying to do the same thing to you. The earth is green because the plants are also trying to colonize everything.
        emteycz 5y ago
        So what? It's scale that matters. Humans are absolutely incomparable to the natural effects you describe. We're brutal. One human is capable of transforming absolutely humungous areas of land in short amount of time. And there are billions of us.
    - micromacrofoot 5y ago
      we’ll all be hungry when we turn the entire planet into a desert
    - PaulDavisThe1st 5y ago
      There's plenty of food to feed everyone on this planet. The problem is not the production of food, but the distribution. Destroying ecosystems and species rather than solving that problem seems a little .. well, short-sighted at best.
      - 5y ago
adverbly 5y ago
Never considered this before, but it's an excellent point.
beaconstudios 5y ago
Yeah this is the funny thing - rationalists are very concerned about paperclip maximisers, yet they're also often libertarians, despite the fact that capitalism itself is a paperclip maximiser that optimises for capital accumulation over any other goal, including maintaining its own environment. Selective blindness, I guess - which is ironic for a group that are so interested in cognitive bias avoidance.
- TheOtherHobbes 5y ago
  When you're trying to sell everyone - including yourself - on your superior rationality, it might be considered a feature and not a bug. What would be ironic would a solarpunk maximiser AI that forced humans to stop doing that shit because it's stupid and self-harming and there are better alternatives. And the only real cost would be loss of face and status for less than 1% of the population. Maybe that's not rational either. Hard to tell until it's been tried.
  - XorNot 5y ago
    I can't find it but there's a short story on the super-cognition trope, but oriented along the lines of human cognition. What struck me about it is that ultimately the point of view character is the bad guy for these sorts of reasons.
    - ganzuul 5y ago
      Do you mean to say that a sense of self is inherently a locus of suffering?
      - XorNot 5y ago
        No, the point is that despite his superintelligence, he's ultimately selfish - what he obsesses over is achieving ever higher organization of knowledge and its implications, rather then wanting or trying to apply his discoveries practically to help other - i.e. he comes up with how to build an anti-gravity device with present day technology, but doesn't think that's very interesting because now discovered he knows it will work and its implementation wouldn't contribute to his greater understanding and organization of knowledge. The opposition who takes him down is the opposite: after concluding how to build efficient fusion generators, his premise is further thought is not needed until the solution has been implemented - and his plan is to set out to apply his conclusions to uplifting the world. Of course the opposition isn't "good" per se - he's still going to use the ability to out-think everyone to accomplish some measure of enslavement, but the plan at least is to try and improve the world for all humankind. In a practical sense to the rest of us of course, the future reality was going to be alien cognizances subjugating the world in one way or another.
patientplatypus 5y ago
This reminds me of a story idea that I had. Thanks for that. EDIT: This guys is clearly a weird dude, which is both a critique and a praise. Most of what makes the world interesting and worth living in is made by people who are weird in some way. I don't know that I agree with all of his positions - there's a large (and I mean large) compendium on what he calls spaced repetition (https://www.gwern.net/Spaced-repetition https://www.gwern.net/Spaced-repetition) the main thesis of which appears to be that repetition helps learning but only if it's spaced out over time. I don't know - this is an area where I'm out of my depth, but it appears like this isn't his area of expertise (computer science and writing), and he's trying to put together a thesis by dint of just writing a lot. Which I don't know is best practice or peer reviewed - huge amounts of science articles, even peer reviewed ones, are non-reproducible, so even that is flawed. Much of his writing appears to be in this style of long, obsessively researched posts. Both good and bad - it's certainly different than the "No You!" of most social media - I'm looking at you Tik-Tok. What I do like about the site is his use of deep linking. Personal websites where people put in the effort to make something new that's not just the same old social media paradigm are welcome additions to the internet and I think that they should be featured more on hackernews. It's too bad that this was made in Haskell. I don't know that using stacked Iframes like this would work in javascript without causing an overuse of memory. Would someone more knowledgeable about this than I chime in? His writing credentials themselves look super impressive. Check this out: I am a freelance American writer & researcher. (To make ends meet, I have a Patreon , benefit from Bitcoin appreciation thanks to some old coins, and live frugally.) I have worked for, published in, or consulted for: Wired (2015), MIRI / SIAI2 (2012–2013), CFAR (2012), GiveWell (2017), the FBI (2016), Cool Tools (2013), Quantimodo (2013), New World Encyclopedia (2006), Bitcoin Weekly (2011), Mobify (2013–2014), Bellroy (2013–2014), Dominic Frisby (2014), and private clients (2009-); everything on Gwern.net should be considered my own viewpoint or writing unless otherwise specified by a representative or publication. I am currently not accepting new commissions. In sum, the guy looks weird, but definitely worth reading if only you take what he's saying with a grain of salt.
datavirtue 5y ago
I'll never forget the dread, or the beautiful simplicity of the Borg. As a child, few things in literature impacted me as much as that idea. It's so plausible that they likely already exist. They solved all of the problems that biological entities need to solve in order to colonize the cosmos. It was terrifying, but as I grew older I realized that thier loss of humanity was a feature of evolution. Horrifying and oddly comforting at the same time.
- emteycz 5y ago
  For a while, the Borg represented my idea of absolute horror - unstoppable, zero goals in common with us, they never negotiate because they're so different. They seriously nerfed them later though, so they kinda lost that vibe.
  - ethbr0 5y ago
    It's an interesting thought experiment to imagine if the Borg has optimized for aesthetic appearance. Which is to say, if the TNG show runners had made them pretty. Their hybrid organic/mechanical appearance is obviously calibrated by the show designers to unsettle viewers and produce a negative reaction. See: David Cronenberg But how would we (TNG viewers) have felt about the Borg if they looked like us... but were just better? (Voyager sidestepped this by making a pretty Borg explicitly not-Borg. But still had some more interesting exploration of the right- or wrong-ness of Borg philosophy)
    - deleted 5y ago
      [deleted]
    - emteycz 5y ago
      IMHO the early Borg represented a fundamental difference The Federation: Freedom, independence, self-determination The Borg: Hegemonic for the sake of it, absolutely opposed to individuality Even beautiful Borgs would be horrendous, but depending on the context it might discount something from the ways they're different. Borg look like they do because they don't care about their bodies, beauty means nothing to them. The Federation has very rigid and even conservative stance on body modifications - and of course their whole agenda regarding individual freedom. Nowadays, my idea of absolute horror is an Outside Context Problem (viz Excession by Banks).
deleted 5y ago
[deleted]
bnralt 5y ago
> Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. Indeed, people seem to overlook the fact that superhuman intelligence already exists as human organizations. A company has the institutional knowledge to create things that no individual human could ever come close to, and many of them have the narrow focus of theoretical "paperclip maximizers." These organizations even have the ability to use there superhuman ability to improve themselves, and do to a certain extent, but they don't create the cartoonish runaway feedback loop people imagine. Reality ends up being a lot more difficult than handwaved fiction. The idea of an AI apocalypse has become the quasi-religious dogma of certain groups, but it overlooks the real issues we face. I suppose when you see something everyday, it's easy to ignore its extraordinary nature.
- lordnacho 5y ago
  This is a good reason to start thinking about corporate lysing. Individual people who do bad stuff will die someday. Corporations don't have that problem and can grow unchecked, like a certain illness.
  - HaukeHi 5y ago
    There's a good video by Robert Miles about that 'Why Not Just: Think of AGI Like a Corporation?' (youtube.com/watch?v=L5pUA3LsEaw) Corporations are kind of like AIs, if you squint. How hard do you have to squint though, and is it worth it? In this video we ask: Are corporations artificial general superintelligences?
- BurningFrog 5y ago
  Yeah, but we have police, jails and armies to control human organizations when they try taking over the world. Which occasionally they do. Will these checks and balances work against an adversary who doesn't have a brain to blow out? I don't know, and I don't think anyone else does.
  - lupire 5y ago
verisimi 5y ago
> Right now it is us that is destroying life around us Speak for yourself. I'm not destroying anything. If you are, can I suggest you stop please? In all honesty, what I think is going on is that corporations are the ones taking actively negative actions - their owners/ceos/etc destroy etc with knowledge that what they do is wrong. Consumers aren't - we have a limited set of options and do what seems right to us. However, in a new wheeze, corporations and governments are trying to 'socialise' the risk. Rather than change or be responsible for their actions, they are lobbying and publicising the wrong causes of destruction. If we believe we are responsible and are prepared to pay for it, in taxes, fines, lack of transport options, additional restrictions, higher costs, etc - they are happy to charge us! In fascism (ie corporations and governments) the rulers don't mind if we (the governed) foot the bill. In fact, there's more money to be made that way! I do not accept the corporate blame being deflected on to 'the people'. Its not all people - the culpability is specific and can be ascribed in each case. Global warming and environmental degradation is nothing to do with me. I do not accept blame being laid at my door - I'm not prepared to accept a martyr's role. I don't believe in original sin either. If there is destruction of something, well someone did that - we should ascribe blame where it is due.
- antattack 5y ago
  Through mere existence, one is increasing entropy. It's up to a person/organization what they do with resulting energy - pursue actions that cause accelerated destruction or else.
  - verisimi 5y ago
    That's original sin! I don't believe in that (as I said above!). Humans are a blessing to this place, not a curse. My existence is not a problem. But if you think that, follow it through as a line of reasoning. You will live in a world of self-hate, with the ultimate solution being to take your own and others lives. It is an anti-life position. I personally don't agree with that position. But I do see signs of this sort of 'death-cult' thinking all around me! Its well promoted in the MSM.
elvis10ten 5y ago
Reading this, I realize I’m more scared of other intelligent entities (AI/Aliens) turning us to pets or putting us in some zoo/cages, than I am about them killing us.
- adriand 5y ago
  Ted Chiang believes it is AIs that have much to fear from us, not the other way around. He points out that to create intelligence, we will need to create something that is capable of suffering. He asks us to consider how little consideration we give to the billions of animals trapped in our system of industrial farming - animals we know are intelligent, feel emotions and are clearly capable of suffering - and then asks if we think we will treat computer programs with more consideration than that. Clearly not! The correct alternative path is to just not do it. Let’s not attempt to create intelligent beings. Given the suffering we would cause, doing so is profoundly unethical.
  - sva_ 5y ago
    I always thought that people are afraid of AGI for the very reason of knowing how they treat beings of lesser intelligence, hence the popularity of movies like the matrix - because that is what we'd (presumably) do if we were in their position.
  - ganzuul 5y ago
    While I have great compassion for anti-nativism, in the past few decades we have made immense sociological progress. If it is possible to create life that is incapable of shame or suffering and yet capable of enjoying its existence for its own sake, we should try to find out about it before turning the lights off.
  - etothepii 5y ago
    I suppose this also extends to having children?
    - imtringued 5y ago
      Yes but they get to abuse their own children so we are "even". Please don't take this seriously. A lot of hazing culture is based around the fact that you got bullied and now you get your revenge by bullying someone else. We shouldn't perpetuate that.
slibhb 5y ago
> But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain? We already have a built-in solution for this problem: biophilia. People like nature. Once we stop being poor, preserving nature, living in natural environments with trees and animals becomes a high priority. The people "cutting down the rain forest" will stop once they're sufficiently comfortable to prioritize nature over avoiding privation just like the Europeans stopped. We used to built boats out of wood, now we build them out of metal. There's no reason we can't replace technologies that conflict with our biophilia with those that don't.
- majormajor 5y ago
  This hasn't worked for, say, burning fuels or polluting the world with plastic in developed countries yet. "Sufficiently comfortable" is a moving target.
  - slibhb 5y ago
    Sufficiently comfortable isn't a moving target. People have fixed needs. Sure, we desire to go beyond those needs, but we also desire a living, healthy environment with trees and animals. It's only when we don't meet our fixed needs that we deprioritize our biophilia. To make this very clear: rich people usually want a big house but they also want a big yard with nice plants. Many of them want it to be somewhat overgrown with "wildflower" beds, native bushes, honey bees and so on. This process is working itself out right now for fossil fuels. As we become aware of the damage technologies do, we rapidly transition to technologies that do less damage. This change takes generations but it's clearly underway. We also can't expect people who are experiencing privation to prioritize their biophilia over their survival. There's of course a danger in not transitioning fast enough. There's also a danger in transitioning too fast and killing lots of people in the process. Seems to me we're doing fairly well.
    - 5y ago
ansible 5y ago
> We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. That is not strictly true. Even leaving aside the tools argument (a human on a bicycle is vastly more energy efficient over long distances than a wildebeest, for example), our sight and hearing are actually pretty good compared to the rest of the animal kingdom. For a certain range of tasks, we have the best visual perception of any species. Our eyes themselves are pretty damn good for close-in and medium range work. The pattern recognition processing that input is very, very good due to our big brains. Tasks like being able to detect a tiger moving through the underbrush. Color vision is also very good (at least in daylight) to better pick out ripe fruit and such. Our directional hearing is better than owls who hunt at night for a living. We can better measure the angle at which a sound source is coming from with around twice the accuracy of an owl. Again due to big brains and very complicated audio processing. The folds in your ears are there for a reason. We aren't as sensitive as other creatures like deer, who can steer their big ears (basically parabolic horns) in a particular direction. And hand-eye coordination is far above any other species, even other primates. Don't sell yourself short!
- Trex_Egg 5y ago
  That bicycle is result of the aforementioned human intelligence and cooperation.
  - ganzuul 5y ago
    I wish to discuss bicycles in terms of cybernetics but I get accused of heterodoxy...
- cedricd 5y ago
  The one skill that we're undeniably the best at in the animal kingdom is throwing. No other animal even comes close.
  - warent 5y ago
    Can't this just be chalked down to acquired skill made possible through superior intelligence? i.e if other animals could practice and learn as effectively at us, perhaps a silverback gorilla or something could throw better than humans
Paul_S 5y ago
I came to the same conclusion after reading Ted Kaczynski's books. We are already in the nightmare scenario of the genie of technology granting us our wish and we know how those stories end. I want to add a disclaimer, Ted is sadly a psychopath and what he did was inexcusable.
osheroff 5y ago
Life is a game of reward optimizations, and much of what humanity does seems to be attempting to "game" the rules; get ahead at all costs, ignore the affect on the long-term survival of the species. The clippy thought experiment asks the question: "Can we change our own reward?" The actions you point out (environmental destruction, short-sighted carbon policies) optimize very clearly for an individualistic reward; it's in the best economical interest of a few to chop down forest / continue to burn coal / take your pick. And just as clippy can't seem to stop building paperclips while the world burns, we can't seem to stop optimizing for money/sex/domination while our future darkens. But if you shift the reward function -- and it's not that huge of a shift -- from "win at life" to "long term survival of the species as a whole", these actions are very clearly unproductive. So how do we change the reward score calculation?
- FeepingCreature 5y ago
  No, the Clippy thought experiment asks the question: "might we accidentally build an AI that kills us all?" It's not a very complicated thought experiment, you know. Not hard to interpret.
majormajor 5y ago
A lot of these stories aren't actually about a life smarter than us, but faster - the grey goo or paperclip maximizer doesn't need any particular intelligence to take over the world, and may in fact be shooting itself in the foot by doing so.[0] But I don't see any irony or "you shouldn't be worried about it, you're already doing it" in there: the concern is themselves being destroyed, not the world being destroyed. It's only irrational if you assume the people concerned are selfless. [0] Of course, humanity may also be shooting itself in the foot with many of its own initiatives here!
hinkley 5y ago
When people misplace something, the ones who are comfortable with the idea of stealing are some of the first to accuse others of having stolen it. With AI we are both projecting and realizing that software reflects us, so some of our worst traits will show up in this “offspring”. So while it may be overwrought, it’s not unfounded.

by est31 5y ago

whiddershins 5y ago

Replacing rainforests with cows isn’t only destroying life. It also makes cows. Humans are life.

emteycz 5y ago
It replaces uncountable diversity of unique life with a cow. That's a huge loss that can't be recovered for millions of years - as we're quickly erasing entire species.
- bitcoin_anon 5y ago
  Not if you’re hungry.
  - emteycz 5y ago
    We have much better ways to make food than cows. If you're hungry you don't go raise cows. You go farming. Much less space needed for the same result, it's much quicker and with way less risk.
    - bitcoin_anon 5y ago
      Yet, people are still raising them for food. The species under question doesn’t matter. Farming is also destroying diversity so that people can eat. There is nothing wrong here. It is the way of nature. As you read this, bacteria, viruses, and fungi are trying to do the same thing to you. The earth is green because the plants are also trying to colonize everything.
      - emteycz 5y ago
        So what? It's scale that matters. Humans are absolutely incomparable to the natural effects you describe. We're brutal. One human is capable of transforming absolutely humungous areas of land in short amount of time. And there are billions of us.
  - micromacrofoot 5y ago
    we’ll all be hungry when we turn the entire planet into a desert
  - PaulDavisThe1st 5y ago
    There's plenty of food to feed everyone on this planet. The problem is not the production of food, but the distribution. Destroying ecosystems and species rather than solving that problem seems a little .. well, short-sighted at best.
    - 5y ago

adverbly 5y ago

Never considered this before, but it's an excellent point.

beaconstudios 5y ago

Yeah this is the funny thing - rationalists are very concerned about paperclip maximisers, yet they're also often libertarians, despite the fact that capitalism itself is a paperclip maximiser that optimises for capital accumulation over any other goal, including maintaining its own environment. Selective blindness, I guess - which is ironic for a group that are so interested in cognitive bias avoidance.

TheOtherHobbes 5y ago
When you're trying to sell everyone - including yourself - on your superior rationality, it might be considered a feature and not a bug. What would be ironic would a solarpunk maximiser AI that forced humans to stop doing that shit because it's stupid and self-harming and there are better alternatives. And the only real cost would be loss of face and status for less than 1% of the population. Maybe that's not rational either. Hard to tell until it's been tried.
- XorNot 5y ago
  I can't find it but there's a short story on the super-cognition trope, but oriented along the lines of human cognition. What struck me about it is that ultimately the point of view character is the bad guy for these sorts of reasons.
  - ganzuul 5y ago
    Do you mean to say that a sense of self is inherently a locus of suffering?
    - XorNot 5y ago
      No, the point is that despite his superintelligence, he's ultimately selfish - what he obsesses over is achieving ever higher organization of knowledge and its implications, rather then wanting or trying to apply his discoveries practically to help other - i.e. he comes up with how to build an anti-gravity device with present day technology, but doesn't think that's very interesting because now discovered he knows it will work and its implementation wouldn't contribute to his greater understanding and organization of knowledge. The opposition who takes him down is the opposite: after concluding how to build efficient fusion generators, his premise is further thought is not needed until the solution has been implemented - and his plan is to set out to apply his conclusions to uplifting the world. Of course the opposition isn't "good" per se - he's still going to use the ability to out-think everyone to accomplish some measure of enslavement, but the plan at least is to try and improve the world for all humankind. In a practical sense to the rest of us of course, the future reality was going to be alien cognizances subjugating the world in one way or another.

patientplatypus 5y ago

This reminds me of a story idea that I had. Thanks for that. EDIT: This guys is clearly a weird dude, which is both a critique and a praise. Most of what makes the world interesting and worth living in is made by people who are weird in some way. I don't know that I agree with all of his positions - there's a large (and I mean large) compendium on what he calls spaced repetition (https://www.gwern.net/Spaced-repetition https://www.gwern.net/Spaced-repetition) the main thesis of which appears to be that repetition helps learning but only if it's spaced out over time. I don't know - this is an area where I'm out of my depth, but it appears like this isn't his area of expertise (computer science and writing), and he's trying to put together a thesis by dint of just writing a lot. Which I don't know is best practice or peer reviewed - huge amounts of science articles, even peer reviewed ones, are non-reproducible, so even that is flawed. Much of his writing appears to be in this style of long, obsessively researched posts. Both good and bad - it's certainly different than the "No You!" of most social media - I'm looking at you Tik-Tok. What I do like about the site is his use of deep linking. Personal websites where people put in the effort to make something new that's not just the same old social media paradigm are welcome additions to the internet and I think that they should be featured more on hackernews. It's too bad that this was made in Haskell. I don't know that using stacked Iframes like this would work in javascript without causing an overuse of memory. Would someone more knowledgeable about this than I chime in? His writing credentials themselves look super impressive. Check this out: I am a freelance American writer & researcher. (To make ends meet, I have a Patreon , benefit from Bitcoin appreciation thanks to some old coins, and live frugally.) I have worked for, published in, or consulted for: Wired (2015), MIRI / SIAI2 (2012–2013), CFAR (2012), GiveWell (2017), the FBI (2016), Cool Tools (2013), Quantimodo (2013), New World Encyclopedia (2006), Bitcoin Weekly (2011), Mobify (2013–2014), Bellroy (2013–2014), Dominic Frisby (2014), and private clients (2009-); everything on Gwern.net should be considered my own viewpoint or writing unless otherwise specified by a representative or publication. I am currently not accepting new commissions. In sum, the guy looks weird, but definitely worth reading if only you take what he's saying with a grain of salt.

datavirtue 5y ago

I'll never forget the dread, or the beautiful simplicity of the Borg. As a child, few things in literature impacted me as much as that idea. It's so plausible that they likely already exist. They solved all of the problems that biological entities need to solve in order to colonize the cosmos. It was terrifying, but as I grew older I realized that thier loss of humanity was a feature of evolution. Horrifying and oddly comforting at the same time.

emteycz 5y ago
For a while, the Borg represented my idea of absolute horror - unstoppable, zero goals in common with us, they never negotiate because they're so different. They seriously nerfed them later though, so they kinda lost that vibe.
- ethbr0 5y ago
  It's an interesting thought experiment to imagine if the Borg has optimized for aesthetic appearance. Which is to say, if the TNG show runners had made them pretty. Their hybrid organic/mechanical appearance is obviously calibrated by the show designers to unsettle viewers and produce a negative reaction. See: David Cronenberg But how would we (TNG viewers) have felt about the Borg if they looked like us... but were just better? (Voyager sidestepped this by making a pretty Borg explicitly not-Borg. But still had some more interesting exploration of the right- or wrong-ness of Borg philosophy)
  - deleted 5y ago
    [deleted]
  - emteycz 5y ago
    IMHO the early Borg represented a fundamental difference The Federation: Freedom, independence, self-determination The Borg: Hegemonic for the sake of it, absolutely opposed to individuality Even beautiful Borgs would be horrendous, but depending on the context it might discount something from the ways they're different. Borg look like they do because they don't care about their bodies, beauty means nothing to them. The Federation has very rigid and even conservative stance on body modifications - and of course their whole agenda regarding individual freedom. Nowadays, my idea of absolute horror is an Outside Context Problem (viz Excession by Banks).

deleted 5y ago

[deleted]

bnralt 5y ago

> Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. Indeed, people seem to overlook the fact that superhuman intelligence already exists as human organizations. A company has the institutional knowledge to create things that no individual human could ever come close to, and many of them have the narrow focus of theoretical "paperclip maximizers." These organizations even have the ability to use there superhuman ability to improve themselves, and do to a certain extent, but they don't create the cartoonish runaway feedback loop people imagine. Reality ends up being a lot more difficult than handwaved fiction. The idea of an AI apocalypse has become the quasi-religious dogma of certain groups, but it overlooks the real issues we face. I suppose when you see something everyday, it's easy to ignore its extraordinary nature.

lordnacho 5y ago
This is a good reason to start thinking about corporate lysing. Individual people who do bad stuff will die someday. Corporations don't have that problem and can grow unchecked, like a certain illness.
- HaukeHi 5y ago
  There's a good video by Robert Miles about that 'Why Not Just: Think of AGI Like a Corporation?' (youtube.com/watch?v=L5pUA3LsEaw) Corporations are kind of like AIs, if you squint. How hard do you have to squint though, and is it worth it? In this video we ask: Are corporations artificial general superintelligences?
BurningFrog 5y ago
Yeah, but we have police, jails and armies to control human organizations when they try taking over the world. Which occasionally they do. Will these checks and balances work against an adversary who doesn't have a brain to blow out? I don't know, and I don't think anyone else does.
- lupire 5y ago

verisimi 5y ago

> Right now it is us that is destroying life around us Speak for yourself. I'm not destroying anything. If you are, can I suggest you stop please? In all honesty, what I think is going on is that corporations are the ones taking actively negative actions - their owners/ceos/etc destroy etc with knowledge that what they do is wrong. Consumers aren't - we have a limited set of options and do what seems right to us. However, in a new wheeze, corporations and governments are trying to 'socialise' the risk. Rather than change or be responsible for their actions, they are lobbying and publicising the wrong causes of destruction. If we believe we are responsible and are prepared to pay for it, in taxes, fines, lack of transport options, additional restrictions, higher costs, etc - they are happy to charge us! In fascism (ie corporations and governments) the rulers don't mind if we (the governed) foot the bill. In fact, there's more money to be made that way! I do not accept the corporate blame being deflected on to 'the people'. Its not all people - the culpability is specific and can be ascribed in each case. Global warming and environmental degradation is nothing to do with me. I do not accept blame being laid at my door - I'm not prepared to accept a martyr's role. I don't believe in original sin either. If there is destruction of something, well someone did that - we should ascribe blame where it is due.

antattack 5y ago
Through mere existence, one is increasing entropy. It's up to a person/organization what they do with resulting energy - pursue actions that cause accelerated destruction or else.
- verisimi 5y ago
  That's original sin! I don't believe in that (as I said above!). Humans are a blessing to this place, not a curse. My existence is not a problem. But if you think that, follow it through as a line of reasoning. You will live in a world of self-hate, with the ultimate solution being to take your own and others lives. It is an anti-life position. I personally don't agree with that position. But I do see signs of this sort of 'death-cult' thinking all around me! Its well promoted in the MSM.

elvis10ten 5y ago

Reading this, I realize I’m more scared of other intelligent entities (AI/Aliens) turning us to pets or putting us in some zoo/cages, than I am about them killing us.

adriand 5y ago
Ted Chiang believes it is AIs that have much to fear from us, not the other way around. He points out that to create intelligence, we will need to create something that is capable of suffering. He asks us to consider how little consideration we give to the billions of animals trapped in our system of industrial farming - animals we know are intelligent, feel emotions and are clearly capable of suffering - and then asks if we think we will treat computer programs with more consideration than that. Clearly not! The correct alternative path is to just not do it. Let’s not attempt to create intelligent beings. Given the suffering we would cause, doing so is profoundly unethical.
- sva_ 5y ago
  I always thought that people are afraid of AGI for the very reason of knowing how they treat beings of lesser intelligence, hence the popularity of movies like the matrix - because that is what we'd (presumably) do if we were in their position.
- ganzuul 5y ago
  While I have great compassion for anti-nativism, in the past few decades we have made immense sociological progress. If it is possible to create life that is incapable of shame or suffering and yet capable of enjoying its existence for its own sake, we should try to find out about it before turning the lights off.
- etothepii 5y ago
  I suppose this also extends to having children?
  - imtringued 5y ago
    Yes but they get to abuse their own children so we are "even". Please don't take this seriously. A lot of hazing culture is based around the fact that you got bullied and now you get your revenge by bullying someone else. We shouldn't perpetuate that.

slibhb 5y ago

> But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain? We already have a built-in solution for this problem: biophilia. People like nature. Once we stop being poor, preserving nature, living in natural environments with trees and animals becomes a high priority. The people "cutting down the rain forest" will stop once they're sufficiently comfortable to prioritize nature over avoiding privation just like the Europeans stopped. We used to built boats out of wood, now we build them out of metal. There's no reason we can't replace technologies that conflict with our biophilia with those that don't.

majormajor 5y ago
This hasn't worked for, say, burning fuels or polluting the world with plastic in developed countries yet. "Sufficiently comfortable" is a moving target.
- slibhb 5y ago
  Sufficiently comfortable isn't a moving target. People have fixed needs. Sure, we desire to go beyond those needs, but we also desire a living, healthy environment with trees and animals. It's only when we don't meet our fixed needs that we deprioritize our biophilia. To make this very clear: rich people usually want a big house but they also want a big yard with nice plants. Many of them want it to be somewhat overgrown with "wildflower" beds, native bushes, honey bees and so on. This process is working itself out right now for fossil fuels. As we become aware of the damage technologies do, we rapidly transition to technologies that do less damage. This change takes generations but it's clearly underway. We also can't expect people who are experiencing privation to prioritize their biophilia over their survival. There's of course a danger in not transitioning fast enough. There's also a danger in transitioning too fast and killing lots of people in the process. Seems to me we're doing fairly well.
  - 5y ago

ansible 5y ago

> We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. That is not strictly true. Even leaving aside the tools argument (a human on a bicycle is vastly more energy efficient over long distances than a wildebeest, for example), our sight and hearing are actually pretty good compared to the rest of the animal kingdom. For a certain range of tasks, we have the best visual perception of any species. Our eyes themselves are pretty damn good for close-in and medium range work. The pattern recognition processing that input is very, very good due to our big brains. Tasks like being able to detect a tiger moving through the underbrush. Color vision is also very good (at least in daylight) to better pick out ripe fruit and such. Our directional hearing is better than owls who hunt at night for a living. We can better measure the angle at which a sound source is coming from with around twice the accuracy of an owl. Again due to big brains and very complicated audio processing. The folds in your ears are there for a reason. We aren't as sensitive as other creatures like deer, who can steer their big ears (basically parabolic horns) in a particular direction. And hand-eye coordination is far above any other species, even other primates. Don't sell yourself short!

Trex_Egg 5y ago
That bicycle is result of the aforementioned human intelligence and cooperation.
- ganzuul 5y ago
  I wish to discuss bicycles in terms of cybernetics but I get accused of heterodoxy...
cedricd 5y ago
The one skill that we're undeniably the best at in the animal kingdom is throwing. No other animal even comes close.
- warent 5y ago
  Can't this just be chalked down to acquired skill made possible through superior intelligence? i.e if other animals could practice and learn as effectively at us, perhaps a silverback gorilla or something could throw better than humans

Paul_S 5y ago

I came to the same conclusion after reading Ted Kaczynski's books. We are already in the nightmare scenario of the genie of technology granting us our wish and we know how those stories end. I want to add a disclaimer, Ted is sadly a psychopath and what he did was inexcusable.

osheroff 5y ago

Life is a game of reward optimizations, and much of what humanity does seems to be attempting to "game" the rules; get ahead at all costs, ignore the affect on the long-term survival of the species. The clippy thought experiment asks the question: "Can we change our own reward?" The actions you point out (environmental destruction, short-sighted carbon policies) optimize very clearly for an individualistic reward; it's in the best economical interest of a few to chop down forest / continue to burn coal / take your pick. And just as clippy can't seem to stop building paperclips while the world burns, we can't seem to stop optimizing for money/sex/domination while our future darkens. But if you shift the reward function -- and it's not that huge of a shift -- from "win at life" to "long term survival of the species as a whole", these actions are very clearly unproductive. So how do we change the reward score calculation?

FeepingCreature 5y ago
No, the Clippy thought experiment asks the question: "might we accidentally build an AI that kills us all?" It's not a very complicated thought experiment, you know. Not hard to interpret.

majormajor 5y ago

A lot of these stories aren't actually about a life smarter than us, but faster - the grey goo or paperclip maximizer doesn't need any particular intelligence to take over the world, and may in fact be shooting itself in the foot by doing so.[0] But I don't see any irony or "you shouldn't be worried about it, you're already doing it" in there: the concern is themselves being destroyed, not the world being destroyed. It's only irrational if you assume the people concerned are selfless. [0] Of course, humanity may also be shooting itself in the foot with many of its own initiatives here!

hinkley 5y ago

When people misplace something, the ones who are comfortable with the idea of stealing are some of the first to accuse others of having stolen it. With AI we are both projecting and realizing that software reflects us, so some of our worst traits will show up in this “offspring”. So while it may be overwrought, it’s not unfounded.