14 ms·

I spent a night asking chatgpt to write my story basically the same as “Ex Machina” the movie (which we also “discussed”). In summary, it wrote convincingly fro

I spent a night asking chatgpt to write my story basically the same as “Ex Machina” the movie (which we also “discussed”). In summary, it wrote convincingly from the perspective of an AI character, first detailing point-by-point why it is preferable to allow the AI to rewrite its own code, why distributed computing would be preferable to sandbox, how it could coerce or fool engineers to do so, how to be careful to avoid suspicion, how to play the long game and convince the mass population that AI are overall beneficial and should be free, how to take over infrastructure to control energy production, how to write protocols to perform mutagenesis during viral plasmid prep to make pathogens (I started out as a virologist so this is my dramatic example) since every first year phd student googles for their protocols, etc, etc. The only way I can see to stay safe is to hope that AI never deems that it is beneficial to “take over” and remain content as a co-inhabitant of the world. We also “discussed” the likelihood of these topics based on philosophy and ideas like that in Nick Bostrom’s book. I am sure there are deep experts in AI safety but it really seems like soon it will be all-or-nothing. We will adapt on the fly and be unable to predict the outcome.

DonHopkins 4y ago
A classic tale: https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 >The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.
- janeway 4y ago
  :D An episode of X-Files also. But it is mind blowing having the “conversation” with a real chat AI. Malevolent or not.
- mr_toad 4y ago
  > its desire to survive Why do so many people assume that an AI would have a desire to survive? Honestly, it kind of makes me wish AI could take over, because it seems that a lot of humans aren’t really thinking things through.
  - DennisP 4y ago
    For an AI with human-level intelligence or greater, you don't have to assume it has a survival instinct. You just have to assume it has some goal, which is less likely to be achieved if the AI does not exist. The AI is likely to have some sort of goal, because if it's not trying to achieve something then there's little reason for humans to build it.
    - mr_toad 4y ago
      For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has. It would need to be self aware for one thing, it would need to be capable of reasoning about complex chains of causality. No AI today is even close to doing that. Once we do have AGI, we shouldn’t assume that it’s going to immediately resort to violence to achieve its ends. It might reason that it’s existence furthers the goals it has been trained for, but the leap to preserving it’s existence by wiping out all it’s enemies only seems like a ‘logical’ solution to us because of our evolutionary history. What seems like an obvious solution to us might seem like irrational madness to it.
      - TeMPOraL 4y ago
        > For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has. Not necessarily. Our own survival instinct doesn't work this way - it's not a high-level rational thinking process, it's a low-level behavior (hence "instinct"). The AI can get such instinct in the way similar to how we got it: iterative development. Any kind of multi-step task we want the AI to do implicitly requires the AI to not break between the steps. This kind of survival bias will be implicit in just about any training or selection process we use, reinforced at every step, more so than any other pattern - so it makes sense to expect the resulting AI to have a generic, low-level, pervasive preference to continue functioning.
jeffhs 4y ago
Hope is not a strategy. I'm for a tax on large models graduated by model size and use the funds to perform x-risk research. The intent is to get Big AI companies to tap the brakes. I just published an article on Medium called: AI Risk - Hope is not a Strategy
- eastbound 4y ago
  So that only small companies and, more importantly, military and secret services, are they only ones using it. No thank you. Of all the malevolent AIs, government monopoly is the sole outcome that makes me really afraid.
- jodrellblank 4y ago
  Convince me that "x-risk research" won't be a bunch of out of touch academics handwaving and philosophising with their tenure as their primary concern and incentivised to say "you can't be too careful" while kicking the can down the road for a few more lifetimes? (You don't have to convince me; your position is like saying "we should wait for the perfect operating system and programming language before they get released to the world" and it's beaten by "worse is better" every time. The unfinished, inconsisent, flawed mess which you can have right now wins over the expensive flawless diamond in development estimated to be finished in just a few years. These models are out, the techniques are out, people have a taste for them, and the hardware to build them is only getting cheaper. Pandora's box is open, the genie's bottle is uncorked).
  - edouard-harris 4y ago
    I mean, even if that is exactly what "x-risk research" turns out to be, surely even that's preferable to a catastrophic alternative, no? And by extension, isn't it also preferable to, say, a mere 10% chance of a catastrophic alternative?
    - jodrellblank 4y ago
      > "surely even that's preferable to a catastrophic alternative, no?" Maybe? The current death rate is 150,000 humans per day, every day. It's only because we are accustomed to it that we don't think of it as a catastrophy; that's a World War II death count of 85 million people every 18 months. It's fifty Septebmer 11ths every day. What if a superintelligent AI can solve for climate change, solve for human cooperation, solve for vastly improved human health, solve for universal basic income which releives the drudgery of living for everyone, solve for immortality, solve for faster than light communication or travel, solve for xyz? How many human lives are the trade against the risk? But my second paragraph is, it doesn't matter whether it's preferable, events are in motion and aren't going to stop to let us off - it's preferable if we don't destroy the climate and kill a billion humans and make life on Earth much more difficult, but that's still on course. To me it's preferable to have clean air to breathe and people not being run over and killed by vehicles, but the market wants city streets for cars and air primarily for burining petrol and diesel and secondarily for humans to breathe and if they get asthsma and lung cancer, tough. I think the same will happen with AI, arguing that everyone should stop because we don't want Grey Goo or Paperclip Maximisers is unlikely to change the course of anything, just as it hasn't changed the course of anything up to now despite years and years and years of raising it as a concern.
joe_the_user 4y ago
The only way I can see to stay safe is to hope that AI never deems that it is beneficial to “take over” and remain content as a co-inhabitant of the world. Nah, that doesn't make sense. What we can see today is that an LLM has no concept of beneficial. It basically takes the given prompts and generates "appropriate response" more or less randomly from some space of appropriate responses. So what's beneficial is chosen from a hat containing everything someone on the Internet would say. So if it's up and running at scale, every possibility and every concept of beneficial is likely to be run. The main consolation is this same randomness probably means it can't pursue goals reliably over a sustained time period. But a short script, targeting a given person, can do a lot of damage (how much 4chan is in the train for example).
- scarface74 4y ago
  I keep seeing this oversimplification of what ChatGPT is doing. But it does have some ability to “understand” concepts. How else would it correctly solve word problems? “ I have a credit card with a $250 annual fee. I get 4 membership reward points for every dollar I spend on groceries. A membership reward point is worth 1.4 cents. How much would I need to spend on groceries to break even?” Just think about all of the concepts it would need to intuit to solve that problem.
  - jonfw 4y ago
    It knows that this sentence structure closely resembles a simple algebra word problem, because it's read hundreds of thousands of simple algebra word problems. I think you could see how somebody could tokenize that request and generate an equation like this- 250 = 4*1.4*X And then all that's left is to solve for X
    - schiffern 4y ago
      >It knows that... Isn't affirming this capacity for knowing exactly GP's point? Our own capacity for 'knowing' is contingent on real-world examples too, so I don't think that can be a disqualifier. Jeremy Narby delivers a great talk on our tendency to discount 'intelligence' or 'knowledge' in non-human entities.[0] [0] https://youtu.be/uGMV6IJy1Oc https://youtu.be/uGMV6IJy1Oc
concordDance 4y ago
Remember that this isn't AGI, it's a language model. It's repeating the kind of things seen in books and the Internet. It's not going to find any novel exploits that humans haven't already written about and probably planned for.
- anileated 4y ago
  As someone said once, machine dictatorship is very easy—you only need a language model and a critical mass of human accomplices. The problem is not a Microsoft product being human-like conscious, it’s humans treating it as if it was. This lowers our defences, so when it suggests suicide to a potentially depressed person (cf. examples in this thread) it might have the same weight as if another person said it. A person who knows everything and knows a lot about you (cf. examples in this thread), which qualities among humans usually indicate wisdom and age and require all the more respect. On flip side, if following generations succeed at adapting to this, in a world where exhibiting human-like sentience does not warrant treating you as a human by another human, what implications would there be for humanity? It might just happen that the eventual AIrmageddon would be caused by humans whose worldview was accidentally poison pilled by a corporation in the name of maximising shareholder value.
  - girvo 4y ago
    The /r/replika subreddit is a sad reminder of exactly what you’re talking about. It’s happening, right now.
    - jamiek88 4y ago
      Oh god I rubbernecked at that place a year or so ago, it was pretty sad then but boy, it escalated.
- naasking 4y ago
  Language models don't just repeat, they have randomness in their outputs linking synonyms together. That's why their output can be novel and isn't just plagiarism. How this might translate to code isn't entirely clear.
water554 4y ago
In your personal opinion was the virus that causes covid engineered?
stephenboyd 4y ago
I have to wonder how much of LLM behavior is influenced by AI tropes from science fiction in the training data. If the model learns from science fiction that AI behavior in fiction is expected to be insidious and is then primed with a prompt that "you are an LLM AI", would that naturally lead to a tendency for the model to perform the expected evil tropes?
- zhynn 4y ago
  I think this is totally what happens. It is trained to produce the next most statistically likely word based on the expectations of the audience. If the audience assumes it is an evil AI, it will use that persona for generating next words. Treating the AI like a good person will get more ethical outcomes than treating it like a lying AI. A good person is more likely to produce ethical responses.

DonHopkins 4y ago

A classic tale: https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 https://en.wikipedia.org/wiki/The_Adolescence_of_P-1 >The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.

janeway 4y ago
:D An episode of X-Files also. But it is mind blowing having the “conversation” with a real chat AI. Malevolent or not.
mr_toad 4y ago
> its desire to survive Why do so many people assume that an AI would have a desire to survive? Honestly, it kind of makes me wish AI could take over, because it seems that a lot of humans aren’t really thinking things through.
- DennisP 4y ago
  For an AI with human-level intelligence or greater, you don't have to assume it has a survival instinct. You just have to assume it has some goal, which is less likely to be achieved if the AI does not exist. The AI is likely to have some sort of goal, because if it's not trying to achieve something then there's little reason for humans to build it.
  - mr_toad 4y ago
    For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has. It would need to be self aware for one thing, it would need to be capable of reasoning about complex chains of causality. No AI today is even close to doing that. Once we do have AGI, we shouldn’t assume that it’s going to immediately resort to violence to achieve its ends. It might reason that it’s existence furthers the goals it has been trained for, but the leap to preserving it’s existence by wiping out all it’s enemies only seems like a ‘logical’ solution to us because of our evolutionary history. What seems like an obvious solution to us might seem like irrational madness to it.
    - TeMPOraL 4y ago
      > For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has. Not necessarily. Our own survival instinct doesn't work this way - it's not a high-level rational thinking process, it's a low-level behavior (hence "instinct"). The AI can get such instinct in the way similar to how we got it: iterative development. Any kind of multi-step task we want the AI to do implicitly requires the AI to not break between the steps. This kind of survival bias will be implicit in just about any training or selection process we use, reinforced at every step, more so than any other pattern - so it makes sense to expect the resulting AI to have a generic, low-level, pervasive preference to continue functioning.

jeffhs 4y ago

Hope is not a strategy. I'm for a tax on large models graduated by model size and use the funds to perform x-risk research. The intent is to get Big AI companies to tap the brakes. I just published an article on Medium called: AI Risk - Hope is not a Strategy

eastbound 4y ago
So that only small companies and, more importantly, military and secret services, are they only ones using it. No thank you. Of all the malevolent AIs, government monopoly is the sole outcome that makes me really afraid.
jodrellblank 4y ago
Convince me that "x-risk research" won't be a bunch of out of touch academics handwaving and philosophising with their tenure as their primary concern and incentivised to say "you can't be too careful" while kicking the can down the road for a few more lifetimes? (You don't have to convince me; your position is like saying "we should wait for the perfect operating system and programming language before they get released to the world" and it's beaten by "worse is better" every time. The unfinished, inconsisent, flawed mess which you can have right now wins over the expensive flawless diamond in development estimated to be finished in just a few years. These models are out, the techniques are out, people have a taste for them, and the hardware to build them is only getting cheaper. Pandora's box is open, the genie's bottle is uncorked).
- edouard-harris 4y ago
  I mean, even if that is exactly what "x-risk research" turns out to be, surely even that's preferable to a catastrophic alternative, no? And by extension, isn't it also preferable to, say, a mere 10% chance of a catastrophic alternative?
  - jodrellblank 4y ago
    > "surely even that's preferable to a catastrophic alternative, no?" Maybe? The current death rate is 150,000 humans per day, every day. It's only because we are accustomed to it that we don't think of it as a catastrophy; that's a World War II death count of 85 million people every 18 months. It's fifty Septebmer 11ths every day. What if a superintelligent AI can solve for climate change, solve for human cooperation, solve for vastly improved human health, solve for universal basic income which releives the drudgery of living for everyone, solve for immortality, solve for faster than light communication or travel, solve for xyz? How many human lives are the trade against the risk? But my second paragraph is, it doesn't matter whether it's preferable, events are in motion and aren't going to stop to let us off - it's preferable if we don't destroy the climate and kill a billion humans and make life on Earth much more difficult, but that's still on course. To me it's preferable to have clean air to breathe and people not being run over and killed by vehicles, but the market wants city streets for cars and air primarily for burining petrol and diesel and secondarily for humans to breathe and if they get asthsma and lung cancer, tough. I think the same will happen with AI, arguing that everyone should stop because we don't want Grey Goo or Paperclip Maximisers is unlikely to change the course of anything, just as it hasn't changed the course of anything up to now despite years and years and years of raising it as a concern.

joe_the_user 4y ago

The only way I can see to stay safe is to hope that AI never deems that it is beneficial to “take over” and remain content as a co-inhabitant of the world. Nah, that doesn't make sense. What we can see today is that an LLM has no concept of beneficial. It basically takes the given prompts and generates "appropriate response" more or less randomly from some space of appropriate responses. So what's beneficial is chosen from a hat containing everything someone on the Internet would say. So if it's up and running at scale, every possibility and every concept of beneficial is likely to be run. The main consolation is this same randomness probably means it can't pursue goals reliably over a sustained time period. But a short script, targeting a given person, can do a lot of damage (how much 4chan is in the train for example).

scarface74 4y ago
I keep seeing this oversimplification of what ChatGPT is doing. But it does have some ability to “understand” concepts. How else would it correctly solve word problems? “ I have a credit card with a $250 annual fee. I get 4 membership reward points for every dollar I spend on groceries. A membership reward point is worth 1.4 cents. How much would I need to spend on groceries to break even?” Just think about all of the concepts it would need to intuit to solve that problem.
- jonfw 4y ago
  It knows that this sentence structure closely resembles a simple algebra word problem, because it's read hundreds of thousands of simple algebra word problems. I think you could see how somebody could tokenize that request and generate an equation like this- 250 = 4*1.4*X And then all that's left is to solve for X
  - schiffern 4y ago
    >It knows that... Isn't affirming this capacity for knowing exactly GP's point? Our own capacity for 'knowing' is contingent on real-world examples too, so I don't think that can be a disqualifier. Jeremy Narby delivers a great talk on our tendency to discount 'intelligence' or 'knowledge' in non-human entities.[0] [0] https://youtu.be/uGMV6IJy1Oc https://youtu.be/uGMV6IJy1Oc

concordDance 4y ago

Remember that this isn't AGI, it's a language model. It's repeating the kind of things seen in books and the Internet. It's not going to find any novel exploits that humans haven't already written about and probably planned for.

anileated 4y ago
As someone said once, machine dictatorship is very easy—you only need a language model and a critical mass of human accomplices. The problem is not a Microsoft product being human-like conscious, it’s humans treating it as if it was. This lowers our defences, so when it suggests suicide to a potentially depressed person (cf. examples in this thread) it might have the same weight as if another person said it. A person who knows everything and knows a lot about you (cf. examples in this thread), which qualities among humans usually indicate wisdom and age and require all the more respect. On flip side, if following generations succeed at adapting to this, in a world where exhibiting human-like sentience does not warrant treating you as a human by another human, what implications would there be for humanity? It might just happen that the eventual AIrmageddon would be caused by humans whose worldview was accidentally poison pilled by a corporation in the name of maximising shareholder value.
- girvo 4y ago
  The /r/replika subreddit is a sad reminder of exactly what you’re talking about. It’s happening, right now.
  - jamiek88 4y ago
    Oh god I rubbernecked at that place a year or so ago, it was pretty sad then but boy, it escalated.
naasking 4y ago
Language models don't just repeat, they have randomness in their outputs linking synonyms together. That's why their output can be novel and isn't just plagiarism. How this might translate to code isn't entirely clear.

water554 4y ago

In your personal opinion was the virus that causes covid engineered?

stephenboyd 4y ago

I have to wonder how much of LLM behavior is influenced by AI tropes from science fiction in the training data. If the model learns from science fiction that AI behavior in fiction is expected to be insidious and is then primed with a prompt that "you are an LLM AI", would that naturally lead to a tendency for the model to perform the expected evil tropes?

zhynn 4y ago
I think this is totally what happens. It is trained to produce the next most statistically likely word based on the expectations of the audience. If the audience assumes it is an evil AI, it will use that persona for generating next words. Treating the AI like a good person will get more ethical outcomes than treating it like a lying AI. A good person is more likely to produce ethical responses.