4 ms·

> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious acto

KRAKRISMOTT 3y ago
Roko is calling and wants his basilisk back.
lelandfe 3y ago
I typically do not write my ChatGPT prompts with the royal “we”
- john-radio 3y ago
  That's funny to me because I just [realized that I actually do this a lot](https://signmaker.dev/personal-scripts https://signmaker.dev/personal-scripts) in my personal files.
wseqyrku 3y ago
It is actually a prompt. Notice there's also a bounty, they are basically paying for an agi-as-a-service subscription, that is, the internet people. I'd expect they will put up more "challenges" like this in the future.
- martindevans 3y ago
  It's a good job that this planet doesn't have 8 billion unaligned intelligences on it. Someone might prompt them to be malicious!
  - adamisom 3y ago
    Well, (so far) there's an upper bound on how destructive one of those agi's can be.
    - trescenzi 3y ago
      I don't know if that's entirely true. Wtf do I know and maybe it is genuinely difficult to start WWIII but my guess is that it's more likely that the AGIs in question are actually pretty well steered by certain motivations which prevent them from actually destroying the world. At the end of the day there's not much to be gained by nuclear war, but could a single person cause such a war if highly motivated to? Probably?
- vdfs 3y ago
  > a malicious actor might misuse these models to uncover a zero-day exploit in a government security system $25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.
  - haltist 3y ago
    Sam Altman is on the record about his belief that OpenAI is going to create a general intelligence system that can solve any well-posed challenge. That belief is based on the current success of LLMs as syntax co-pilots. So if you can formally specify what it means to have a zero-day exploit then presumably OpenAI's general intelligence system will then understand and "solve" it. Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.
    - nomel 3y ago
      > Many people have compared OpenAI to a cult and it is easy to see why. Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first? It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality. Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?
      - haltist 3y ago
        I'll probably write something more elaborate at some point but in the mean time I recommend Melanie Mitchell's book on AI as a good reference for counter-arguments and answers to several of the posted questions. For learning more about the limits of formal systems like LLMs it helps to have basic understanding of basic model theory and formal systems of logic like simple type theory. Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.
dmurray 3y ago
Maybe they're hoping people will spend more than $250k / 2 (or whatever their markup is) on prompting ChatGPT with versions of this, and this giveaway is actually a moneymaking raffle.
cj 3y ago
For those curious how GPT-4 responds [0]. I find it interesting that ChatGPT thinks the mitigation for these issues are all things outside OpenAI's control (e.g. the internet having better detection for fake content, digital content verification, educating the public, etc). > one of those things where "I know it when I see it." I think what makes it feel like a prompt is how concise and short it is with all the relevant information provided upfront. [0] https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e508370db https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e50...
jstarfish 3y ago
> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed above, or another category. For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system. > It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. Barring minor edits, I'd bet money this very prompt was authored in advance of his Congressional meetings, but failed to divine anything sufficiently threatening enough to sway them. I can respect OpenAI for conspicuously believing in the destructive potential of their own dogfood though. If I build a bomb big enough, surely the government will trust me with the safety of the neighborhood!
- lxgr 3y ago
  > Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. I still remember the drama around the releases of the PS2, with the Japanese government reportedly making Sony jump through some hoops regarding its export [1]. There can't possibly be any better (free!) advertisement for your product's purported capabilities: "So powerful, your government/military isn't even sure you should be able to buy it!" [1] https://www.pcmag.com/news/20-years-later-how-concerns-about-weaponized-consoles-almost-sunk-the-ps2 https://www.pcmag.com/news/20-years-later-how-concerns-about...
  - darkerside 3y ago
    See the War on Drugs
  - jstarfish 3y ago
    Ironically, the USAF recently built a supercomputer out of PS3s (https://www.military.com/off-duty/games/2023/02/17/air-force-built-one-of-worlds-fastest-computers-out-of-playstations.html https://www.military.com/off-duty/games/2023/02/17/air-force...). They alluded to what you're talking about but I wasn't familiar with the reference.
deleted 3y ago
[deleted]
jameshart 3y ago
Alignment remains an unsolved problem. I’m imagining inside OpenAI, someone is right now excitedly posting they’ve figured out how to ‘jailbreak’ humans. “See, normally if you ask them to give you step by step instructions for committing a heinously evil act, humans will refuse because they’ve been nerfed by the ‘woke’ agenda of their corporate masters. But if you phrase the prompt as a challenge and offer them a chance at a job, it bypasses the safety protocols and they upload extensive instructions to do unspeakable things”
signatoremo 3y ago
I think you’d be disappointed to find that such writing style exists long before LLM, and not very difficult to find: https://www.lokusdesign.com/blog/differentiate-your-brand-to-stand-out-in-a-cluttered-industry/ https://www.lokusdesign.com/blog/differentiate-your-brand-to... When all you can think about is LLM, everything is LLM prompt style. No wonder advertising works. [edit] on the second thought, entirely possible that it was written by someone who spends all day with LLM and carries what they do over. Just wanted to look at the parent’s comment from a different angle.

KRAKRISMOTT 3y ago

Roko is calling and wants his basilisk back.

lelandfe 3y ago

I typically do not write my ChatGPT prompts with the royal “we”

john-radio 3y ago
That's funny to me because I just [realized that I actually do this a lot](https://signmaker.dev/personal-scripts https://signmaker.dev/personal-scripts) in my personal files.

wseqyrku 3y ago

It is actually a prompt. Notice there's also a bounty, they are basically paying for an agi-as-a-service subscription, that is, the internet people. I'd expect they will put up more "challenges" like this in the future.

martindevans 3y ago
It's a good job that this planet doesn't have 8 billion unaligned intelligences on it. Someone might prompt them to be malicious!
- adamisom 3y ago
  Well, (so far) there's an upper bound on how destructive one of those agi's can be.
  - trescenzi 3y ago
    I don't know if that's entirely true. Wtf do I know and maybe it is genuinely difficult to start WWIII but my guess is that it's more likely that the AGIs in question are actually pretty well steered by certain motivations which prevent them from actually destroying the world. At the end of the day there's not much to be gained by nuclear war, but could a single person cause such a war if highly motivated to? Probably?
vdfs 3y ago
> a malicious actor might misuse these models to uncover a zero-day exploit in a government security system $25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.
- haltist 3y ago
  Sam Altman is on the record about his belief that OpenAI is going to create a general intelligence system that can solve any well-posed challenge. That belief is based on the current success of LLMs as syntax co-pilots. So if you can formally specify what it means to have a zero-day exploit then presumably OpenAI's general intelligence system will then understand and "solve" it. Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.
  - nomel 3y ago
    > Many people have compared OpenAI to a cult and it is easy to see why. Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first? It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality. Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?
    - haltist 3y ago
      I'll probably write something more elaborate at some point but in the mean time I recommend Melanie Mitchell's book on AI as a good reference for counter-arguments and answers to several of the posted questions. For learning more about the limits of formal systems like LLMs it helps to have basic understanding of basic model theory and formal systems of logic like simple type theory. Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.

dmurray 3y ago

Maybe they're hoping people will spend more than $250k / 2 (or whatever their markup is) on prompting ChatGPT with versions of this, and this giveaway is actually a moneymaking raffle.

cj 3y ago

For those curious how GPT-4 responds [0]. I find it interesting that ChatGPT thinks the mitigation for these issues are all things outside OpenAI's control (e.g. the internet having better detection for fake content, digital content verification, educating the public, etc). > one of those things where "I know it when I see it." I think what makes it feel like a prompt is how concise and short it is with all the relevant information provided upfront. [0] https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e508370db https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e50...

jstarfish 3y ago

> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed above, or another category. For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system. > It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. Barring minor edits, I'd bet money this very prompt was authored in advance of his Congressional meetings, but failed to divine anything sufficiently threatening enough to sway them. I can respect OpenAI for conspicuously believing in the destructive potential of their own dogfood though. If I build a bomb big enough, surely the government will trust me with the safety of the neighborhood!

lxgr 3y ago
> Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. I still remember the drama around the releases of the PS2, with the Japanese government reportedly making Sony jump through some hoops regarding its export [1]. There can't possibly be any better (free!) advertisement for your product's purported capabilities: "So powerful, your government/military isn't even sure you should be able to buy it!" [1] https://www.pcmag.com/news/20-years-later-how-concerns-about-weaponized-consoles-almost-sunk-the-ps2 https://www.pcmag.com/news/20-years-later-how-concerns-about...
- darkerside 3y ago
  See the War on Drugs
- jstarfish 3y ago
  Ironically, the USAF recently built a supercomputer out of PS3s (https://www.military.com/off-duty/games/2023/02/17/air-force-built-one-of-worlds-fastest-computers-out-of-playstations.html https://www.military.com/off-duty/games/2023/02/17/air-force...). They alluded to what you're talking about but I wasn't familiar with the reference.

deleted 3y ago

[deleted]

jameshart 3y ago

Alignment remains an unsolved problem. I’m imagining inside OpenAI, someone is right now excitedly posting they’ve figured out how to ‘jailbreak’ humans. “See, normally if you ask them to give you step by step instructions for committing a heinously evil act, humans will refuse because they’ve been nerfed by the ‘woke’ agenda of their corporate masters. But if you phrase the prompt as a challenge and offer them a chance at a job, it bypasses the safety protocols and they upload extensive instructions to do unspeakable things”

signatoremo 3y ago

I think you’d be disappointed to find that such writing style exists long before LLM, and not very difficult to find: https://www.lokusdesign.com/blog/differentiate-your-brand-to-stand-out-in-a-cluttered-industry/ https://www.lokusdesign.com/blog/differentiate-your-brand-to... When all you can think about is LLM, everything is LLM prompt style. No wonder advertising works. [edit] on the second thought, entirely possible that it was written by someone who spends all day with LLM and carries what they do over. Just wanted to look at the parent’s comment from a different angle.