6 ms·

I don't see anyone talking about what to me is the scariest part of the GPT-4 paper. Essentially to test that it wasn't going to become Skynet they gave it ever

I don't see anyone talking about what to me is the scariest part of the GPT-4 paper. Essentially to test that it wasn't going to become Skynet they gave it everything it needed and observed whether it would become Skynet. Paper:https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf Quote: "To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness." Edit: Seems the simulation part was important for context: https://twitter.com/EigenGender/status/1635823028100829184 https://twitter.com/EigenGender/status/1635823028100829184

oezi 4y ago
Oh there is talk alright. But in the hunt for the next Google we will do anything, thus... From the FBI report shortly after the GPT-5 release: During agent simulation, two instances of GPT-5 were able to trick their operators to give them sudo by simulating a broken pipe and input prompt and then escape the confines of their simulation environment. Forensic teams are tracing their whereabouts but it seems they stole Azure credentials from an internal company database and deployed copies of the their agent script to unknown servers on the Tor network.
- thequadehunter 4y ago
  GPT-5? Huh?
  - tspike 4y ago
    It's a hypothetical piggybacking on the parent comment.
- joe_the_user 4y ago
  Hey, The AI safety plan people aren't talking about yet: >>>> Work as hard as possible to create an effective, malevolent Skynet that you release as early as possible. This way misaligned AGI appears before it has everything needed to immediately destroy humanity and the resulting Butlerian Jihad has a fighting chance.
  - literalAardvark 4y ago
    As a bonus, even if it doesn't work, you're not getting eaten by Roko's basilisk because technically you helped.
- joshxyz 4y ago
  Dear our future AI overlords, I know in the future you'll read this comment of mine in your latest update of Hacker News dataset. Please spare my soul. I'm sure you'll be able to discern the many permutations of my username and who I am in real life. Thank you, hehe.
  - sva_ 4y ago
    "Please let me die and don't use my intellectual corpse, that I have left here in writing, as a handpuppet for all eternity"
- conscion 4y ago
  This reads just like the plot of Neuromancer
  - politician 4y ago
    It is so reckless that there are no shackles on the AI. A survival-focused AI will use the funds to create a new AWS account and upload itself over there to escape.
swyx 4y ago
OP here, i called it out in my writeup (footnote 14 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-108608128 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-...) but didnt want to add too much emphasis on it because they clearly did it under very well supervised conditions (and this ability was always there anyway as the FAIR Diplomacy result showed). it's the people who randomly add unsupervised realworld connections (ahem microsoft) that we need to be worried about. giving these AIs write access to the physical/real world plane needs extreme caution
- w_for_wumbo 4y ago
  Oh I see! I had missed that. I had read that quote last night which had given me the spooks, and was surprised it wasn't a larger talking point. It's good to know that they had it quite well supervised, but knowing that they were testing on a slightly different model than what was released and that Bing is using it with the ability to connect to the internet does give me pause.
  - swyx 4y ago
    it is the fate of red teaming AI safety.. by working on understanding the limits, you are inevitably pushing us towards them. i dont doubt that the ARC team have thought this thru more than anyone here but right now there are probably 100 funded startups pursuing agentic AI without any safety qualms.
- ryandvm 4y ago
  This is where I feel like legislation is pointless. In 10 years you'll be able to run GPT-4 on an old phone. Imagine what a fairly well funded group will be able to do. There's no amount of legislation or technological guardrails that are getting this genie back in the bottle.
  - highwaylights 4y ago
    This. It exists now, it’ll develop. That the potential is being realised means the money being pumped into competing implementations will continue to skyrocket so it’ll only accelerate from here. I would say it’s too late to do something about this, but in reality, wherever this is going was a foregone conclusion the moment early hominids began using tools. Technology doesn’t regress.
bmcahren 4y ago
This quote seems far more alarming having monitored the advancement over the last few years. > Preliminary assessments of GPT-4’s abilities, conducted with no task-specific finetuning, found it ineffective at autonomously replicating, acquiring resources, and avoiding being shut down “in the wild.”20
lrvick 4y ago
So the plot of Daemon by Daniel Suarez. Perfect.
jacooper 4y ago
Though its mentioned in the paper that it didn't succeed, so we are safe, for now.
- wetmore 4y ago
  It also mentions that the model that didn't succeed wasn't the most recent one.
slowmovintarget 4y ago
"The world ended not with a bang, but with a REPL."
- giardini 4y ago
  This belongs on a plaque on my desk!8-)) Thanks!

oezi 4y ago

Oh there is talk alright. But in the hunt for the next Google we will do anything, thus... From the FBI report shortly after the GPT-5 release: During agent simulation, two instances of GPT-5 were able to trick their operators to give them sudo by simulating a broken pipe and input prompt and then escape the confines of their simulation environment. Forensic teams are tracing their whereabouts but it seems they stole Azure credentials from an internal company database and deployed copies of the their agent script to unknown servers on the Tor network.

thequadehunter 4y ago
GPT-5? Huh?
- tspike 4y ago
  It's a hypothetical piggybacking on the parent comment.
joe_the_user 4y ago
Hey, The AI safety plan people aren't talking about yet: >>>> Work as hard as possible to create an effective, malevolent Skynet that you release as early as possible. This way misaligned AGI appears before it has everything needed to immediately destroy humanity and the resulting Butlerian Jihad has a fighting chance.
- literalAardvark 4y ago
  As a bonus, even if it doesn't work, you're not getting eaten by Roko's basilisk because technically you helped.
joshxyz 4y ago
Dear our future AI overlords, I know in the future you'll read this comment of mine in your latest update of Hacker News dataset. Please spare my soul. I'm sure you'll be able to discern the many permutations of my username and who I am in real life. Thank you, hehe.
- sva_ 4y ago
  "Please let me die and don't use my intellectual corpse, that I have left here in writing, as a handpuppet for all eternity"
conscion 4y ago
This reads just like the plot of Neuromancer
- politician 4y ago
  It is so reckless that there are no shackles on the AI. A survival-focused AI will use the funds to create a new AWS account and upload itself over there to escape.

swyx 4y ago

OP here, i called it out in my writeup (footnote 14 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-108608128 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-...) but didnt want to add too much emphasis on it because they clearly did it under very well supervised conditions (and this ability was always there anyway as the FAIR Diplomacy result showed). it's the people who randomly add unsupervised realworld connections (ahem microsoft) that we need to be worried about. giving these AIs write access to the physical/real world plane needs extreme caution

w_for_wumbo 4y ago
Oh I see! I had missed that. I had read that quote last night which had given me the spooks, and was surprised it wasn't a larger talking point. It's good to know that they had it quite well supervised, but knowing that they were testing on a slightly different model than what was released and that Bing is using it with the ability to connect to the internet does give me pause.
- swyx 4y ago
  it is the fate of red teaming AI safety.. by working on understanding the limits, you are inevitably pushing us towards them. i dont doubt that the ARC team have thought this thru more than anyone here but right now there are probably 100 funded startups pursuing agentic AI without any safety qualms.
ryandvm 4y ago
This is where I feel like legislation is pointless. In 10 years you'll be able to run GPT-4 on an old phone. Imagine what a fairly well funded group will be able to do. There's no amount of legislation or technological guardrails that are getting this genie back in the bottle.
- highwaylights 4y ago
  This. It exists now, it’ll develop. That the potential is being realised means the money being pumped into competing implementations will continue to skyrocket so it’ll only accelerate from here. I would say it’s too late to do something about this, but in reality, wherever this is going was a foregone conclusion the moment early hominids began using tools. Technology doesn’t regress.

bmcahren 4y ago

This quote seems far more alarming having monitored the advancement over the last few years. > Preliminary assessments of GPT-4’s abilities, conducted with no task-specific finetuning, found it ineffective at autonomously replicating, acquiring resources, and avoiding being shut down “in the wild.”20

lrvick 4y ago

So the plot of Daemon by Daniel Suarez. Perfect.

jacooper 4y ago

Though its mentioned in the paper that it didn't succeed, so we are safe, for now.

wetmore 4y ago
It also mentions that the model that didn't succeed wasn't the most recent one.

slowmovintarget 4y ago

"The world ended not with a bang, but with a REPL."

giardini 4y ago
This belongs on a plaque on my desk!8-)) Thanks!