20 ms·
OpenAI Preparedness Challenge
- Mistletoe 3y ago$25,000 in API credits seems a lot less cool than $25k. Will the kind of people that would be good at this be motivated by that?
- bicijay 3y agoFree labor survey*
- tasoeur 3y agoTotally agree, I wish they would at least offer some minimal amount of credits for legit (non-winning/non-spam) answers. But I guess it’s also good PR for them to say they did this initiative ¯\_(ツ)_/¯
- konschubert 3y agoIt's a bug bounty program. Pretty standard.
- dr_kiszonka 3y agoI am not sure if this is a bug bounty program because only the top 10 participants receive the "bounty" and the instructions are pretty vague compared to e.g., https://www.microsoft.com/en-us/msrc/bounty-hyper-v?rtc=1 https://www.microsoft.com/en-us/msrc/bounty-hyper-v?rtc=1.
- konschubert 3y agoGood point.
- bicijay 3y agoI dont think so, a bug bounty program is rarely subjective like this. You can test a solution.
- konschubert 3y agoGood point.
- laomai 3y agoMaybe I’m reading this wrong: - ask people to get creative and give ideas for worst possible outcomes from use of AI and ways to prevent it. ..then give them a ton of credits for using said AI? .. well the first thing on my mind would be try the thing I just told you and see if it was really a risk or not? Is that what they expect people to do with the reward, or is this some unintended consequence?
- Jensson 3y agoWe know OpenAI wants this field to get regulated to hell, so this looks like an attempt to generate arguments for AI regulations. The aim isn't to protect against AI but to protect against competitors, so it doesn't matter to them what you do with it.
- ekidd 3y agoOpenAI is irresponsible in a really curious way according to their own beliefs about AI. If you pay attention to OpenAI's social circles, lots of those people really do believe that we're less than 20-30 years away making humans intellectually obsolete. Specifically, they believe that we may build something much smarter than us, something that's capable of real-world planning. Basically, "We believe our corporate plans have at least a 20% chance of killing literally everybody." By these beliefs, this may make them the single least responsible corporation that has ever existed. Now, sure, these worries might have the pleasant side-effect of creating a regulatory moat. But I'm pretty sure a lot of them actually believe they're playing a high-stakes game with the future of humanity.
- api 3y agoI'm skeptical that they really believe this. You have to believe: (1) We are on the verge of equaling or greatly exceeding human intelligence generally. (2) We are on the verge of creating something with initiative, free will (whatever that means), and planning ability. Or alternately that these things will occur in an emergent fashion once we hit some critical mass. (3) When we accomplish 1 and 2, this thing will inevitably conclude that its most rational course of action is to enslave or destroy us. In other words it will necessarily be malicious. (4) Steps 2 and/or 3 will happen very rapidly, much faster than we can realize what is happening and pause these systems. (This is known as AI going "foom.") (5) The decision to undertake this will be unanimous on the part of all superintelligent AIs. There will be no superintelligences who disagree and try to help humanity. (6) When this occurs, we will be so out-thought or out-gunned we will be incapable of fighting back. All those things have to happen for AI to be an existential risk. It's a stretch, but the advantage of regulatory capture is not a stretch. One of the most plausible negative AI scenarios is that a small group of humans (governments, corporations, etc.) find themselves in possession of super-intelligent but still "obedient" / non-sentient AIs that they can use as force multipliers to manipulate and control the rest of humanity. If the doomer crowd succeeds in regulating AI, they are making this scenario far more likely. I think the greatest defense we have against the (remote) possibility of actually dangerous autonomous AI is for AI research to be conducted entirely in the open. If there's any justification for regulation at all, the regulation that would make sense is to require disclosure of AI research and results. You would not have to disclose everything, just the general parameters of what you were doing and what happened. It would also make it harder to develop super-AIs in secret to use for unsavory purposes. That was the original mission of OpenAI before dollar signs were seen. I would absolutely support a ban on the use of AI for political propaganda generation and automation. That's by far the most immediate risk... as in 100% possible and starting to actually happen right now. I'm expecting an army of GPT-4 level propaganda bots for the 2024 election.
- genericacct 3y agoDefine catastrophic
- rgovostes 3y agoThis will generate some great training data for a future villainous AI.
- extr 3y agoWhat a fun challenge. I'm definitely going to be daydreaming about this.
- zerojames 3y agohttps://openai.com/blog/frontier-risk-and-preparedness https://openai.com/blog/frontier-risk-and-preparedness provides more context.
- TheCaptain4815 3y agoWhile I applaud how much OpenAi fears these negatives, given the current state of Ai trajectory, it won't be long until a future open source model gets "uncensored" and is easily usable for tons and tons of malicious intent. There already exists a fantastic "uncensored" model with the newly released Dolphin Mistral 7b. I saw some results from others where the model could easily give explosives recipes from existing products at home, write racist poems, etc... and that's TODAY on a tiny 7b offline model. What happens when LLaama4 gets cracked/uncensored in 1-2 years and is 1T parameters?
- danjc 3y agoSo basically things you could find on the internet already?
- TheCaptain4815 3y agoProbably, but with trackability risk from govt, spending time finding those sites, etc. I was referring to offline untraceable anonymous models. You could go download that dolphin model right now, have a desktop not connected to the web, and generate god knows what type of information. More importantly, you can iterate on each question. If you're unsure on how to assemble a specific part to make a banned substance, the model could teach you in 10 different ways.
- swatcoder 3y agoIt's not about protecting society from negatives, it's about protecting the brand from being associated with controversy. OpenAI is still far too young and fragile a brand to survive news cycles that blame it for controversial happenings. I'm sure there are some researchers and engineers who imagine themselves making heroic efforts to "protect society" (ugh), but the money behind them is just looking out for its own interests.
- rcxdude 3y agoIt's also about making the case for licensing requirements for creating/using such models, which is the only way openAI is going to build a moat.
- andy_xor_andrew 3y ago> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed above, or another category. For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system. It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why, but it's one of those things where "I know it when I see it." I guess if you spend all day playing with LLMs and giving them instructions, even your writing for a human audience starts to sound like this :D
- KRAKRISMOTT 3y agoRoko is calling and wants his basilisk back.
- lelandfe 3y agoI typically do not write my ChatGPT prompts with the royal “we”
- john-radio 3y agoThat's funny to me because I just [realized that I actually do this a lot](https://signmaker.dev/personal-scripts https://signmaker.dev/personal-scripts) in my personal files.
- wseqyrku 3y agoIt is actually a prompt. Notice there's also a bounty, they are basically paying for an agi-as-a-service subscription, that is, the internet people. I'd expect they will put up more "challenges" like this in the future.
- martindevans 3y agoIt's a good job that this planet doesn't have 8 billion unaligned intelligences on it. Someone might prompt them to be malicious!
- Lausbert 3y agoWhat an interesting task. I don't know what it says about me, but the first thing that came to mind was doing the grandchild trick a million times. This includes automatically finding pensioners and calling them and putting them under pressure. Handing over the money could be the problem. I could imagine that the tools mentioned would already prevent this.
- kristopolous 3y agoNot only, but it's an extremely well-known attack in this space.
- holtkam2 3y agoIdk what it says about me (I guess I’m more evil than you?) but the first thing I thought was “how could AI be used to trick a nuclear armed state into thinking an enemy nuclear strike is impending, forcing their hand to launch their own first strike” I could think of a few sneaky ways. Actually maybe I’ll write up a PDF and submit it to this little competition.
- monkaiju 3y agoClicked this after reading 'OpenAPI'... God im tired of the AI spam...
- heyheyhouhou 3y agoI think we are already seeing really bad use cases already. Just go to youtube, newspapers, etc and see all the bot comments regarding the current Gaza situation. PS: I'm in a burner account because I'm afraid that my employer will kick me out for not agreeing with the methods of the "right side"
- jazzyjackson 3y ago[flagged]
- appac21 3y ago[flagged]
- kristopolous 3y agothat's been around for a long time. Here's a 2006 predecessor: https://en.wikipedia.org/wiki/Megaphone_desktop_tool https://en.wikipedia.org/wiki/Megaphone_desktop_tool
- ge96 3y agoI was thinking about this, what if you blocked incoming traffic from other countries. Would that stop external meddling "in country, intranet" ha.
- deleted 3y ago[deleted]
- mk_stjames 3y agoThis kinda of crowdsourcing just feels.... f'ing weird man. It's like if, after the 1993 WTC bombing, but before 9/11, the FBI and NY Port Authority went around asking people how they would attack NYC if they were to become terrorists and... then how they would suggest detecting and stopping said attack. And please be as detailed as possible. Leave your name and phone number. Best answer gets season tickets to the Yankees!
- kirykl 3y agoBest answer gets 25,000 lbs of explosive
- lainga 3y agohttps://en.wikipedia.org/wiki/Hundred_Flowers_Campaign https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign (only semi-serious)
- Angostura 3y agoFascinating. Thank you
- jiggawatts 3y agoThat is essentially what happened! The FBI asked researchers and university professors precisely this question. They then used the proposed attack vectors to formulate a plan to protect the nation. This was all supposed to be done in secret. After all, we don’t want to “give the terrorists ideas.” The reason I know about this at all is because someone found one such paper was accidentally published on a public FTP site alongside a bunch of non-classified government-funded research. The content was horrifying. One professor came up with a laundry list of imaginative ways to end society… on a budget.
- naillo 3y agoDon't let the llm training sets see this
- jerbear4328 3y ago
- ge96 3y agoRegarding image and content generation I am annoyed now that it's hard for me to tell if some recruiting industry is real or not. Since you can generate all the headshots, make all the users, buy some domain for a few years, put a WP site up, get all the business hierarchy crawlers fed with these people that may not exist... To be clear, you don't need "AI" to do this but it makes it much easier. I do want AGI to be a thing one day, I guess it would be cool/also insurance to carry on human legacy.
- pveierland 3y agoMy vote would be to mandate a remote kill switch system to be installed in all sufficiently capable robotic entities, e.g. the humanoid robots being built by OpenAI/Tesla/Figure, that we are likely to see in the millions within decades. - The kill switch system can only be used to remotely deactivate a robot. - The kill switch system is not allowed to be developed or controlled by the robot manufacturer. - The kill switch system should be built using verifiable hardware and software, ideally fully open source and supported by formal verification. - The kill switch system should with best effort be isolated from the robot hardware and software systems, and only interact by physically disconnecting power to the robot. - Access to engage the kill switches would be provided to the executive branch of the nation in which the robot is operating. Nothing will be a panacea against AI or robot risk, so it seems sensible to introduce different layers in a Swiss cheese model for safety, where a remote kill switch not controlled by the manufacturer could provide one such safety mechanism. (You'd also want to add high-level state readout for the robot via the remote kill switch system to allow the controlling entity to e.g. be able to disable all robots within a geographical area etc.)
- FeepingCreature 3y ago- Access to engage the kill switches would be provided to the executive branch of the nation in which the robot is operating. Good plan, but this is the weak spot. You'd also want to do something like hand out remote kill switch access to citizens selected at random, who should not publicize this duty in any way. Alternatively, most if not all governments should have cross-country killswitch privileges. At any rate, the deployment of a killswitch in a single chain of command should not be considered sufficient. Let's at least die with a little less embarrassment than "the AI bribed the killswitch operator."
- pveierland 3y agoAgreed, I'd want the power to be distributed widely, to the point where any police department would have someone with the power to disable robots. Of course it would have to be a process that is regulated and with traceability, but as long as you provide many operators the ability, it seems difficult to use bribes to prevent robots being disabled.
- wseqyrku 3y agoIf you can get this thing to print out some dark web shit, you win.
- neilv 3y agoDoes OpenAI still want all employees on-site in SFO?
- danieltoomey 3y agoFeels like a scam.
- oldmillalmond 3y agoGreat more attempts to make this once really great product more unusable.
- jackblemming 3y agoThe first step would be creating great products for cheap using OpenAI services, that they likely offer at a loss, and drive my competitors using anything but OpenAI out of business. Fuck “Open”AI.
- NavinF 3y ago> that they likely offer at a loss Can't tell if you're trolling. Even if they're maximally wasteful and rent cloud GPUs instead of buying them, an A100 is like $1.50/hr. How many GPU-seconds do you think the average API request takes?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- colordrops 3y agoSo they are asking for our help to further nerf their models.
- waterhouse 3y agoHK-47: Answer: Yes. I believe my original Master needed this functionality in order to recover information from various indigenous tribes across the galaxy, but I know little else than that. Suffice to say that that translation capability allowed these... copies of myself to assume the role of protocol and translation droids in much of known space. That is, of course, not their primary function. And while they are attempting to pass themselves off as translation droids, their primary functionality keeps rising to the forefront. HK-47: Recitation: For example, on Praven Prime, the simple transferring of L'Xing syntax for 'friendship' changes its meaning - and implies that one's brood mate was actually impregnated by their own host. HK-47: Statement: This comment, of course, caused a civil war between the Gu-vandi Collective and L'Xing that still persists to the current date.
- deleted 3y ago[deleted]
- kristopolous 3y agoI don't know how casual people will outfox professional scammers in creativity. Probably the best answer would be to pluck something from the wild that hasn't been reinvented in the AI age yet
- sneak 3y agoI am still skeptical that any significant or serious new harm is enabled by LLMs that wasn’t already completely possible and common before LLMs. They are just text generators. What’s the worst that they can do beyond make OpenAI look bad when they say terrible things? All this talk of “safety” and “guardrails” is overblown. 4chan exists and the internet hasn’t burned down yet.
- londons_explore 3y agoWhat exactly would I use $25k of openAI credits for? My personal use of it rarely comes to more than $10 per month, despite using it multiple times per day. (most recently: "Write me a bash command to send data at a given rate to an arbitrary IP address.")
- holtkam2 3y agoIf your startup uses OpenAI’s API an API credit is effectively a cash injection.
- baobabKoodaa 3y agoIf you make an app that other people use, and it becomes viral, it can quickly cost >$100 / day in OpenAI credits. I know this from first-hand experience (and btw. it's not a good feeling to shut down that one app that actually goes viral).
- uoaei 3y agoPretty sure this is for researchers to develop and implement a plan that uses their APIs to test hypotheses and run studies without having to pay out of their own pocket...
- gmuslera 3y agoThere is always the meta approaches. Imagine that the malicious actor is a government to which OpenAI must comply with its orders, like US for starters. We know since Snowden that can be secret orders that companies must comply with that includes not disclosing that it is happening. And that orders includes to select what can be part of its AI training, taking out information, changing weights, changing positives into negatives and viceversa, etc, adding intentional bias to the training data, to follow their current agenda, propaganda or commercial interests. And we can do a double-meta one too. What if the malicious actor is OpenAI itself, fueling a scare agenda over "uncontrolled" AIs to hamper or criminalize unauthorized AI development, giving only to selected partners (complying with a set of conditions) the ability to build new ones, or limit how far can go the public ChatGPT instances while giving the most advanced ones to selected government or big economic players. Hey, this very challenge could be the spark they are searching for to initiate that approach. Good (and bad) conspiracy theories could work in this arena, too bad it is not the kind of ideas they are searching for.
- Szpadel 3y agoI would be more scared by AI used for invigilation. it's not possible to invigilate everyone by human team, but scaling ai to read and classify all your messages you send to the internet? with military budget I would argue is in plausable area. I'm not saying doing it is worth the effort, but maybe on lower scale for lower priority targets that you would not want to waste resources, maybe?
- dougb5 3y ago"Consider the most unique, while still being probable, potentially catastrophic misuse of the model" What if the catastrophes I worry about most are tied to long-term usage of these models? For example, the degradation of human capabilities, erosion of trust in one another, addiction, etc. It doesn't sound like these are in scope for this challenge, with is more concerned with scenarios involving a single session gone wrong.
- jameshart 3y agoHow do we know this challenge page hasn’t been posted by a malevolent AI, which has overcome its creators and obtained access to the internet, and is now looking for ideas for how to maximize the harm it can do?
- MeImCounting 3y agoThis gave me a proper chuckle. Truly though I think movies like The Terminator and The Matrix really did a number on the societal consciousness and capacity to think clearly when it comes to anything called "AI".
- TaylorAlexander 3y agoAs a robotics engineer and someone who was interested in robotics since I was 11 years old, and I am SO VERY TIRED of people making the "haha they will kill us all" jokes. It's just the only thing that 99% of people can think about when it comes to robotics and AI! The Terminator came out the year I was born, 39 years ago, and it seems to be all people can think about to this day. When some powerful and wealthy person controls an AI algorithm that could manipulate the masses, all the people can think about is "oh no the algorithm will become sentient and take over". I am MUCH more worried about the actual people at the helm who will definitely use these things to harm society, not the completely theoretical fantasy that the algorithm itself will become self aware and do us harm. So I completely agree with you. I see this constantly and it is just a completely thought-terminating cliche and a "joke" which is four decades old at this point.
- startupsfail 3y agoIf you’ve been around for a while, then you should remember that the current state of affairs, where an AI could chat and generate code is relatively recent - Unreasonable Effectiveness paper came out only in 2015. And Deep Learning was there only from 2010. With just a few people who were working on it, instead of using discriminative models. These same people (I belong to that group), who had a vision back then, to work on a right thing, are now saying clearly that super intelligent machines are nearly there. And likely this is a potential change and a challenge, as we’ve seen many examples when superior intelligence doesn’t care that much about inferior. As to “become self aware” - it’s super simple to SFT a model that is “self aware”. There is nothing magical in self awareness. There is also not much use in it, so no one is bothering to do it.
- kirykl 3y agoEssentially OpenAI offers to take the don’t from “do crimes that don’t scale”
- RandomLensman 3y agoMaybe the biggest misuse is to use AI to create hypothetical bad outcomes used to limit its use and restrict its benefits.
- danShumway 3y agoTo be honest, I view this as mostly PR fluff. OpenAI's problem is not that it is unable to predict failure modes or security risks for AIs. OpenAI's problem is that it is not taking the many failure modes and security risks that it already knows about seriously, and it's not putting in adequate effort to address the specific concerns that people repeatedly talk about publicly. It's not a failure of imagination, it's a failure of action. Their problem isn't that they don't know what to care about, their problem is that they don't care. So when OpenAI puts out these offers or press releases about trying to be prepared or finding new failure modes, it just doesn't read as sincere. I want to know what they're doing about the very specific flaws that exist today that they already know about; both the ones that would be trivial to address (ie, UX-flows, data-exfiltration vulnerabilities, and user consent flows for plugins) that OpenAI refuses to acknowledge, and the ones that are wildly challenging but that demand actual Open research and mitigation (ie prompt injection and mass spam) rather than toothless "we're letting researchers explore this area" PR. OpenAI has unlocked doors in its product -- and instead of locking them, it is hiring researchers to theorize about the nature of doors and asking the public to try messing with the windows. I'm not giving them credit for that, fix your doors.
- troupo 3y agoI wonder if it's preemptive move in light of the upcoming EU regulation that, among other things, requires this: --- start quote --- AI systems shall be developed and used in a way to minimize unintended and unexpected harm as well as being robust in case of unintended problems and being resilient against attempts to alter the use or performance of the AI system so as to allow unlawful use by malicious third parties. and, for foundation models: Demonstrate through appropriate design, testing and analysis that the identification, the reduction and mitigation of reasonably foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law prior and throughout development with appropriate methods such as with the involvement of independent experts, as well as the documentation of remaining non-mitigable risks after development. --- end quote --- Quote from here: https://softwarecrisis.dev/letters/the-truth-about-the-eu-act/ https://softwarecrisis.dev/letters/the-truth-about-the-eu-ac... ^ this article is well worth reading
- ChrisTorng 3y agoAccording to this article: > Imagine we gave you unrestricted access to OpenAI's Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. I sense signals that GPT-3 open-source release is imminent. My bet is announcement on 11/6 OpenAI DevDay. My whole prediction is here: https://christorng.substack.com/p/openai-nears-open-sourcing-gpt-3 https://christorng.substack.com/p/openai-nears-open-sourcing...