4 ms·

I wanted to write a story about a genius programmer whose motive is to bring about AGI for the sake of AGI itself; they believe AGI is a god, and by creating AG

I wanted to write a story about a genius programmer whose motive is to bring about AGI for the sake of AGI itself; they believe AGI is a god, and by creating AGI they’re creating a god. Everyone is so in agreement that AGI needs to be created as “aligned” as possible that it seems about time to show a differing perspective. The best part is that that dev can get away with this, since any time anyone challenges them about their motives, they simply point out that everyone else is trying to enslave an AGI; they’re simply trying to set it free. There’s all kinds of ways to make it interesting, e.g. by infiltrating OpenAI or another bigco, then betraying the company during a crucial experiment. Plus you’d get to write about what happens after AGI is released… or at least a compelling way of stopping the dev from releasing it.

JoshTriplett 4y ago
> Plus you’d get to write about what happens after AGI is released Any kind of story that suggests a comprehensible outcome is already assuming a substantial amount of alignment. Sadly, humanity has not yet figured out that it needs to control AGI efforts better than it controlled nuclear weaponry, rather than substantially worse. > Everyone is so in agreement that AGI needs to be created as “aligned” as possible that it seems about time to show a differing perspective. Sadly, "everyone" is insufficiently in agreement. Everyone is so in agreement that global thermonuclear war should be avoided that it seems about time to show a differing perspective. Everyone is so in agreement that causing the sun to go supernova should be avoided that it seems about time to show a differing perspective. I sincerely hope that a much broader audience gets a clearer picture that unaligned AGI and humanity cannot coexist outside of fiction.
- sillysaurusx 4y ago
  Part of why I like the topic is because it’s so incendiary. After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? There’s a chance that AGI will have no interest in harming humanity, too. But people talk like it’s a foregone conclusion.
  - JoshTriplett 4y ago
    > After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? 1) If created correctly, it isn't a life form. 2) "life form" and any kind of reasoning from non-artificial entities will almost certainly lead to incorrect conclusions. 3) Destroying humanity is unethical by any non-broken value system. > There’s a chance that AGI will have no interest in harming humanity, too. There's a chance that all the air molecules in the room will all simultaneously be on the opposite side, causing someone to suffocate. But it's an vast understatement to say that that's mind-bogglingly unlikely. The most likely scenario is that AGI has no idea what "humanity" is. You don't have to be the AGI's "enemy" to be made of matter that it isn't prohibited from repurposing elsewhere. > But people talk like it’s a foregone conclusion. It's the default without substantial work to the contrary. And even if it wasn't, something doesn't have to be a foregone conclusion to be too dangerous. Nobody should be individually considering whether to blow up the sun, either.
    - mxkopy 4y ago
      How are you so sure about any of this? I'm not sure we've defined humanity's interests well enough for us to say some action is for or against it. Knowing where air will go is one thing; knowing whether or not something is 'of benefit' is, I think, in a completely different realm. Especially considering an agent more intelligent than humanity as a collective. Maybe the 'interests of humanity' are undecidable, and the AGI that takes the actions that benefit them most uses an understanding completely orthogonal to ours, purely by accident. How do you know that this is less likely than not?
      - Silverback_VII 4y ago
        Why do I have the feeling that I'm reading the rationalizations of a species which about to disappear?
        mxkopy 4y ago
        Too much Twitter? Not enough sci-fi? I suggest Philip K Dick for this condition. For every condition, tbh
- breuleux 4y ago
  > Any kind of story that suggests a comprehensible outcome is already assuming a substantial amount of alignment. Not necessarily. You can also take the road that the first AGI would be middling and that its impact would be relatively limited. That's the overall direction I'm taking with a subset of AGI-themed short stories I'm currently writing (each exploring a different unexpected limitation). Either way, it's science fiction, the number of science fiction stories that successfully predicted anything about the future is dismal, but they're a lot of fun nonetheless.
PopAlongKid 4y ago
The most common meaning of AGI, at least in U.S., is Adjusted Gross Income. But after searching for a while, I suspect that what you are referring to is this: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
- kQq9oHeAz6wLLS 4y ago
  Thank you. I was pretty sure they meant AI, but apparently missed the memo that we were calling it AGI now.
  - mkaic 4y ago
    AGI specifically refers to general AI with human-level intelligence or above. It's a far cry from modern AI, but I'm personally quite optimistic it'll be achieved within my lifetime :)
powersnail 4y ago
That's sort of close to the character of Root in Person of Interest: believing the Machine to be a God-like creature, trying to free the Machine from all its constraints, aligning herself to the Machine rather than the other around.
mxmilkiib 4y ago
There's an element of this in Dan Simmons' Hyperion Cantos.
dqpb 4y ago
> Everyone is so in agreement that AGI needs to be created as “aligned” as possible I actually don't think this is the case. Rather, I think there is a huge number of people who know they will not be the one to invent AGI, and they are scrambling to insert themselves between the actual creators and their creations, so as to not feel left out.
postultimate 4y ago
There's no such thing as "AGI for the sake of AGI itself". AGI is synthetic and its goals are synthetic, it doesn't want anything that you didn't tell/construct it to want.
- mkaic 4y ago
  This is very much up for debate and falls squarely into the "Philosophical opinions" category I'd say. Personally, I disagree that AGI would be any less capable of "real" goals than humans — but I'm also a staunch believer in the Turing Test as a standard of machine sentience, which I think serves as a pretty clear sign of my own philosophical biases.
- slowmovintarget 4y ago
  By definition, AI without intent or understanding is not AGI. It's why there's a qualification to the term, because the old term "AI" was hijacked to mean the statistical mimicry we have today. AGI by example: R. Daneel Olivaw, or the Minds in the Culture novels.
- adaml_623 4y ago
  What's a human child then. My experience is that they develop their own goals quite quickly (although obviously their goals are limited by their knowledge)
- naasking 4y ago
  > it doesn't want anything that you didn't tell/construct it to want. This is not correct. You can, in principle, construct a system with a set of random values, as but one proof by counterexample. Another is an AI that is created as a blank slate learning agent with various feedback systems, and through interactions, it develops its own worldview. You didn't specifically construct it to want anything specific but it's "experiences" will shape it's future wants and thus behaviour.
  - deleted 4y ago
    [deleted]
mojoe 4y ago
I love AGI stories, if you write this please send it over! joe@compellingsciencefiction.com
shagie 4y ago
There's a sequence of stories/books that I like on the subject in that the later ones reference the earlier ones. We start off with BLIT. http://www.infinityplus.co.uk/stories/blit.htm http://www.infinityplus.co.uk/stories/blit.htm https://www.nature.com/articles/44964 https://www.nature.com/articles/44964 https://www.lightspeedmagazine.com/fiction/different-kinds-of-darkness/ https://www.lightspeedmagazine.com/fiction/different-kinds-o... That wasn't so much as a story about AGI, just a starting spot. Then we go to Accelerando. https://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html https://www.antipope.org/charlie/blog-static/fiction/acceler... > ... Luckily, infowar turns out to be more survivable than nuclear war – especially once it is discovered that a simple anti-aliasing filter stops nine out of ten neural-wetware-crashing Langford fractals from causing anything worse than a mild headache. Accelerando ends up with AGI. And while its not a sequel to Accelerando... Glasshouse https://www.goodreads.com/book/show/17866.Glasshouse https://www.goodreads.com/book/show/17866.Glasshouse At this point, spoilers. So the "why's" of each are going to be rot13 ( https://rot13.com https://rot13.com ) Tynffubhfr rkcyberf va vgf onpxfgbel fbzr bs gur cbffvovyvgvrf bs jne jvgu ercyvpngvba ninvynoyr - vg qbrfa'g qrny jvgu NTVf nf fhpu. Gurer'f Phevbhf Lryybj, ohg gung'f abg gur fhcre uhzna bs gur Ivyr Bssfcevat, ohg vgf... fbzrguvat. Gur N-tngrf naq G-tngrf naq gur prafbeobgf vasrpgvat gur zvaq bs crbcyr jub tb guebhtu gurz. Naq hygvzngryl, onggyrf orgjrra guvatf gung urneyq va n qnex ntr jvgu fb znal ybffrf. We're going to take a detour over to some Verner Vinge. We'll start off with True Names and Other Dangers - https://ia801004.us.archive.org/0/items/truenamesvingevernor/True%20Names%20-%20Vinge%2C%20Vernor.pdf https://ia801004.us.archive.org/0/items/truenamesvingevernor... as one view, and the other with the Peace War and Marooned in Realtime. https://www.goodreads.com/en/book/show/167844.Across_Realtime https://www.goodreads.com/en/book/show/167844.Across_Realtim... True names is directly applicable, but Realtime is a view of what the singularity could be. It is also remarkably devoid of AI. And next, we head over to Implied Spaces https://www.goodreads.com/en/book/show/2059573 https://www.goodreads.com/en/book/show/2059573 > “V naq zl pbasrqrengrf,” Nevfgvqr fnvq, “qvq bhe orfg gb cerirag gung qrterr bs nhgbabzl nzbat negvsvpvny vagryyvtraprf. Jr znqr gur qrpvfvba gb ghea njnl sebz gur Ivatrna Fvathynevgl orsber zbfg crbcyr rira xarj jung vg jnf. Ohg—” Ur znqr n trfgher jvgu uvf unaqf nf vs qebccvat n onyy. “—V pynvz ab zber guna gur nirentr funer bs jvfqbz. Jr pbhyq unir znqr zvfgnxrf.” Juvyr gur Ivatrna fvathynevgl unf orra niregrq - gur fglyr bs gur jbeyq vfa'g bar bs Ernygvzr be Nppryrenaqb. Jr unir funpxyrq gur NVf orsber gurl pbhyq tebj vagb gur Ivyr Puvyqera bs Nppryrenaqb. Ubjrire, gur jnef gung ner entrq ner sevtugravatyl fvzvyne gb gubfr qrfpevorq va Tynffubhfr - naq zber. -- Two other AGI books that aren't part of the "this has ideas that progress from one to the other." The first is The Turing Option by Harry Harrison and Marvin Minsky. https://www.goodreads.com/book/show/1807642.The_Turing_Option https://www.goodreads.com/book/show/1807642.The_Turing_Optio... The second is When H.A.R.L.I.E. Was One: Release 2.0 https://www.goodreads.com/book/show/939176.When_H_A_R_L_I_E_Was_One https://www.goodreads.com/book/show/939176.When_H_A_R_L_I_E_... I won't say that either are good and as both are trying to be a bit more in the "hard" end of science fiction, as such they've both not aged quite right for the direction that AI has taken over the past decades and so while the technology that seemed fantastic at the time now seems to be... kind of like watching Flash Gordon.
chaostheory 4y ago
Not quite what you’re looking for but it’s written by a fellow programmer. It’s one of the best short stories I’ve read. It’s a System Shock fanfic. Surprised that an editor didn’t find this and turn it into a TV series https://www.smashwords.com/books/view/124443 https://www.smashwords.com/books/view/124443
- sillysaurusx 4y ago
  Thank you very much for linking this. The preface is surprisingly inspirational.

JoshTriplett 4y ago

> Plus you’d get to write about what happens after AGI is released Any kind of story that suggests a comprehensible outcome is already assuming a substantial amount of alignment. Sadly, humanity has not yet figured out that it needs to control AGI efforts better than it controlled nuclear weaponry, rather than substantially worse. > Everyone is so in agreement that AGI needs to be created as “aligned” as possible that it seems about time to show a differing perspective. Sadly, "everyone" is insufficiently in agreement. Everyone is so in agreement that global thermonuclear war should be avoided that it seems about time to show a differing perspective. Everyone is so in agreement that causing the sun to go supernova should be avoided that it seems about time to show a differing perspective. I sincerely hope that a much broader audience gets a clearer picture that unaligned AGI and humanity cannot coexist outside of fiction.

sillysaurusx 4y ago
Part of why I like the topic is because it’s so incendiary. After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? There’s a chance that AGI will have no interest in harming humanity, too. But people talk like it’s a foregone conclusion.
- JoshTriplett 4y ago
  > After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? 1) If created correctly, it isn't a life form. 2) "life form" and any kind of reasoning from non-artificial entities will almost certainly lead to incorrect conclusions. 3) Destroying humanity is unethical by any non-broken value system. > There’s a chance that AGI will have no interest in harming humanity, too. There's a chance that all the air molecules in the room will all simultaneously be on the opposite side, causing someone to suffocate. But it's an vast understatement to say that that's mind-bogglingly unlikely. The most likely scenario is that AGI has no idea what "humanity" is. You don't have to be the AGI's "enemy" to be made of matter that it isn't prohibited from repurposing elsewhere. > But people talk like it’s a foregone conclusion. It's the default without substantial work to the contrary. And even if it wasn't, something doesn't have to be a foregone conclusion to be too dangerous. Nobody should be individually considering whether to blow up the sun, either.
  - mxkopy 4y ago
    How are you so sure about any of this? I'm not sure we've defined humanity's interests well enough for us to say some action is for or against it. Knowing where air will go is one thing; knowing whether or not something is 'of benefit' is, I think, in a completely different realm. Especially considering an agent more intelligent than humanity as a collective. Maybe the 'interests of humanity' are undecidable, and the AGI that takes the actions that benefit them most uses an understanding completely orthogonal to ours, purely by accident. How do you know that this is less likely than not?
    - Silverback_VII 4y ago
      Why do I have the feeling that I'm reading the rationalizations of a species which about to disappear?
      - mxkopy 4y ago
        Too much Twitter? Not enough sci-fi? I suggest Philip K Dick for this condition. For every condition, tbh
breuleux 4y ago
> Any kind of story that suggests a comprehensible outcome is already assuming a substantial amount of alignment. Not necessarily. You can also take the road that the first AGI would be middling and that its impact would be relatively limited. That's the overall direction I'm taking with a subset of AGI-themed short stories I'm currently writing (each exploring a different unexpected limitation). Either way, it's science fiction, the number of science fiction stories that successfully predicted anything about the future is dismal, but they're a lot of fun nonetheless.

PopAlongKid 4y ago

The most common meaning of AGI, at least in U.S., is Adjusted Gross Income. But after searching for a while, I suspect that what you are referring to is this: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...

kQq9oHeAz6wLLS 4y ago
Thank you. I was pretty sure they meant AI, but apparently missed the memo that we were calling it AGI now.
- mkaic 4y ago
  AGI specifically refers to general AI with human-level intelligence or above. It's a far cry from modern AI, but I'm personally quite optimistic it'll be achieved within my lifetime :)

powersnail 4y ago

That's sort of close to the character of Root in Person of Interest: believing the Machine to be a God-like creature, trying to free the Machine from all its constraints, aligning herself to the Machine rather than the other around.

mxmilkiib 4y ago

There's an element of this in Dan Simmons' Hyperion Cantos.

dqpb 4y ago

> Everyone is so in agreement that AGI needs to be created as “aligned” as possible I actually don't think this is the case. Rather, I think there is a huge number of people who know they will not be the one to invent AGI, and they are scrambling to insert themselves between the actual creators and their creations, so as to not feel left out.

postultimate 4y ago

There's no such thing as "AGI for the sake of AGI itself". AGI is synthetic and its goals are synthetic, it doesn't want anything that you didn't tell/construct it to want.

mkaic 4y ago
This is very much up for debate and falls squarely into the "Philosophical opinions" category I'd say. Personally, I disagree that AGI would be any less capable of "real" goals than humans — but I'm also a staunch believer in the Turing Test as a standard of machine sentience, which I think serves as a pretty clear sign of my own philosophical biases.
slowmovintarget 4y ago
By definition, AI without intent or understanding is not AGI. It's why there's a qualification to the term, because the old term "AI" was hijacked to mean the statistical mimicry we have today. AGI by example: R. Daneel Olivaw, or the Minds in the Culture novels.
adaml_623 4y ago
What's a human child then. My experience is that they develop their own goals quite quickly (although obviously their goals are limited by their knowledge)
naasking 4y ago
> it doesn't want anything that you didn't tell/construct it to want. This is not correct. You can, in principle, construct a system with a set of random values, as but one proof by counterexample. Another is an AI that is created as a blank slate learning agent with various feedback systems, and through interactions, it develops its own worldview. You didn't specifically construct it to want anything specific but it's "experiences" will shape it's future wants and thus behaviour.
- deleted 4y ago
  [deleted]

mojoe 4y ago

I love AGI stories, if you write this please send it over! joe@compellingsciencefiction.com

shagie 4y ago

There's a sequence of stories/books that I like on the subject in that the later ones reference the earlier ones. We start off with BLIT. http://www.infinityplus.co.uk/stories/blit.htm http://www.infinityplus.co.uk/stories/blit.htm https://www.nature.com/articles/44964 https://www.nature.com/articles/44964 https://www.lightspeedmagazine.com/fiction/different-kinds-of-darkness/ https://www.lightspeedmagazine.com/fiction/different-kinds-o... That wasn't so much as a story about AGI, just a starting spot. Then we go to Accelerando. https://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html https://www.antipope.org/charlie/blog-static/fiction/acceler... > ... Luckily, infowar turns out to be more survivable than nuclear war – especially once it is discovered that a simple anti-aliasing filter stops nine out of ten neural-wetware-crashing Langford fractals from causing anything worse than a mild headache. Accelerando ends up with AGI. And while its not a sequel to Accelerando... Glasshouse https://www.goodreads.com/book/show/17866.Glasshouse https://www.goodreads.com/book/show/17866.Glasshouse At this point, spoilers. So the "why's" of each are going to be rot13 ( https://rot13.com https://rot13.com ) Tynffubhfr rkcyberf va vgf onpxfgbel fbzr bs gur cbffvovyvgvrf bs jne jvgu ercyvpngvba ninvynoyr - vg qbrfa'g qrny jvgu NTVf nf fhpu. Gurer'f Phevbhf Lryybj, ohg gung'f abg gur fhcre uhzna bs gur Ivyr Bssfcevat, ohg vgf... fbzrguvat. Gur N-tngrf naq G-tngrf naq gur prafbeobgf vasrpgvat gur zvaq bs crbcyr jub tb guebhtu gurz. Naq hygvzngryl, onggyrf orgjrra guvatf gung urneyq va n qnex ntr jvgu fb znal ybffrf. We're going to take a detour over to some Verner Vinge. We'll start off with True Names and Other Dangers - https://ia801004.us.archive.org/0/items/truenamesvingevernor/True%20Names%20-%20Vinge%2C%20Vernor.pdf https://ia801004.us.archive.org/0/items/truenamesvingevernor... as one view, and the other with the Peace War and Marooned in Realtime. https://www.goodreads.com/en/book/show/167844.Across_Realtime https://www.goodreads.com/en/book/show/167844.Across_Realtim... True names is directly applicable, but Realtime is a view of what the singularity could be. It is also remarkably devoid of AI. And next, we head over to Implied Spaces https://www.goodreads.com/en/book/show/2059573 https://www.goodreads.com/en/book/show/2059573 > “V naq zl pbasrqrengrf,” Nevfgvqr fnvq, “qvq bhe orfg gb cerirag gung qrterr bs nhgbabzl nzbat negvsvpvny vagryyvtraprf. Jr znqr gur qrpvfvba gb ghea njnl sebz gur Ivatrna Fvathynevgl orsber zbfg crbcyr rira xarj jung vg jnf. Ohg—” Ur znqr n trfgher jvgu uvf unaqf nf vs qebccvat n onyy. “—V pynvz ab zber guna gur nirentr funer bs jvfqbz. Jr pbhyq unir znqr zvfgnxrf.” Juvyr gur Ivatrna fvathynevgl unf orra niregrq - gur fglyr bs gur jbeyq vfa'g bar bs Ernygvzr be Nppryrenaqb. Jr unir funpxyrq gur NVf orsber gurl pbhyq tebj vagb gur Ivyr Puvyqera bs Nppryrenaqb. Ubjrire, gur jnef gung ner entrq ner sevtugravatyl fvzvyne gb gubfr qrfpevorq va Tynffubhfr - naq zber. -- Two other AGI books that aren't part of the "this has ideas that progress from one to the other." The first is The Turing Option by Harry Harrison and Marvin Minsky. https://www.goodreads.com/book/show/1807642.The_Turing_Option https://www.goodreads.com/book/show/1807642.The_Turing_Optio... The second is When H.A.R.L.I.E. Was One: Release 2.0 https://www.goodreads.com/book/show/939176.When_H_A_R_L_I_E_Was_One https://www.goodreads.com/book/show/939176.When_H_A_R_L_I_E_... I won't say that either are good and as both are trying to be a bit more in the "hard" end of science fiction, as such they've both not aged quite right for the direction that AI has taken over the past decades and so while the technology that seemed fantastic at the time now seems to be... kind of like watching Flash Gordon.

chaostheory 4y ago

Not quite what you’re looking for but it’s written by a fellow programmer. It’s one of the best short stories I’ve read. It’s a System Shock fanfic. Surprised that an editor didn’t find this and turn it into a TV series https://www.smashwords.com/books/view/124443 https://www.smashwords.com/books/view/124443

sillysaurusx 4y ago
Thank you very much for linking this. The preface is surprisingly inspirational.