6 ms·
OpenAI Cybersecurity Grant Program
- nynx 3y agoOne of exciting outputs of LLMs will be, I hope, to automatically port large software projects to other languages.
- cj 3y agoFree startup idea: 1. Create something that monitors an app's network connections 2. Use the data to auto-generate integration tests for every endpoint including edge cases, error states, etc 3. Have user validate and accept the integration tests (end there for self-serve tier, and for the enterprise tier...) 4. Rewrite app into another language, framework, or simply refactor code to be cleaner, more consistent, convert callbacks to promises or async/await, etc. ... key step being #3, need a really good test suite to validate the new code works the same as the old.
- tikkun 3y agoLooks like a good initiative. Cybersecurity is certainly one of the near term (now?) AI related risks. This should help encourage some defensive progress.
- photochemsyn 3y agoFunny how it's all going William-Gibson-Sprawl-shaped. The best defensive ice (Intrusion Countermeasures Electronic) was written by AIs who constantly updated it, and the best icebreakers were also written by AI. I'd be very surprised if secretive agencies within the USA, China, Israel, Russia and others weren't going full-bore on the offensive side (which OpenAI says this particular grant program is not involved in, at least, but I wonder what the black-budget NSA-TAO side of things looks like at present).
- drvdevd 3y agoIndeed any AI defense capabilities will necessarily need to include some offensive abilities, which for the public are mostly censored right now (eg try asking for any of the obvious public PoCs which it almost certainly was trained on) I find it to be one of the worst aspects of the current alignment….
- dinvlad 3y agoI don’t think it’s a serious limitation though, since the open-source LLMs like Alpaca/LLaMA have already “won” the mass adoption among engineers, and are only getting better (already better than GPT4 in some areas).
- thevania 3y agothank you for posting this here, wouldn't have found it otherwise
- thevania 3y agowhy exactly did i get a downvote on this? HN is against thanking people for posting stuff?
- chatmasta 3y agoI thought this would be about a bug bounty program, which it isn't, but it did lead me to check if there is an OpenAI bug bounty program, which there is: https://bugcrowd.com/openai https://bugcrowd.com/openai
- VHRanger 3y agoDoes this bug bounty include passthrough issues like the ones here? https://simonwillison.net/2023/May/2/prompt-injection-explained/ https://simonwillison.net/2023/May/2/prompt-injection-explai...
- sebzim4500 3y agoNo, it explicitly does not include prompt injection attacks for obvious reasons.
- VHRanger 3y agoThe obvious reason that they don't want to be responsible for the ensuing clusterfuck?
- buildbot 3y agoLikely because prompt injection is basically instruction injection, aka. asking it to do anything…
- insanitybit 3y agoUnless the prompt somehow bypasses a meaningful security boundary (and not just some best effort moderation) I don't see why they would care at all.
- _23sd 3y agoFor areas that require a great deal of confidence in conclusions, like forensics and incident triage, I can only see generative AI making matters worse. There’s already a ton of over-automation and humans making mistakes because they trust their tools too much. Giving that industry access to more tools that confidently give wrong answers almost seems irresponsible. I am excited for AI-based threat intelligence products, though.
- notif1 3y agoI could see the exact opposite happening. Forensics and IR often require a ton of complex information parsing that an LLM could accomplish very quickly and then point an analyst in the right direction to verify. It's not meant to be a panacea just an accelerant. Threat intel has purported to use AI for ages and I remain unconvinced of its effectiveness. Program analysis to support TI could be exciting however.
- toomuchtodo 3y agoAgree, but having the LLM explain how it arrived at the output will be crucial for provenance and anti-hallucination purposes. Judges don’t care for fabricated evidence or citations.
- chaxor 3y agoThis language bothers me deeply. So many people ask the system itself to describe what it is doing, for which it is absolutely incapable of doing. The system has no access to its internals. Even academics are asking the LLM itself to describe what it's doing like idiots. I know that academia's intellect is slipping due to allowing too many people in, but this type of stupidity is just mind boggling.
- dinvlad 3y agoI like to preserve skepticism here as well, though probably blaming them won’t help the issue. But absolutely agreed, just listened to Gary Marcus’s podcast episode on how LLMs are already capable of making dad jokes, but they can’t do neither “deep” humor generation nor “deep” humor explanation very well, in part because they lack all of the cultural and experiential (“touchy-feely”) context of the humans, and an even bigger part since they are optimized for generation, not explanation, while humans are almost the opposite.
- galacticaactual 3y agoThis is clearly a play for getting access to organizations' logs for them to train their models on - probably in preparation for a future cyber related offering.
- z3phyr 3y agoI miss the 90s security groups which were both funky and respectable and (most importantly) devoid of corporate lingo. Semi underground stuff like Phrack, 2600 and various punky groups who surprisingly did good security research and open to participation.. Today's cutting edge security research feels very silod in the products.
- thelittleone 3y agoOh, the nostalgia... the associated EFNET channels were fun.
- easterncalculus 3y agoI wouldn't even say it's surprisingly good research at this point. A lot of the basics being taught in universities still cite Phrack for example.
- weinzierl 3y agoA bit related: OWASP is working on a Top 10 for large language models. OWASP Top 10 for Large Language Model Applications https://owasp.org/www-project-top-10-for-large-language-model-applications/# https://owasp.org/www-project-top-10-for-large-language-mode... The draft of the Top 10 https://owasp.org/www-project-top-10-for-large-language-model-applications/descriptions/ https://owasp.org/www-project-top-10-for-large-language-mode...
- bink 3y ago> We will grant in increments of $10,000 USD from a fund of $1M USD, in the form of API credits, direct funding and/or equivalents. Does "increments of $10k" mean that a single grant can be greater than $10k? Or are grants limited to that amount?
- ignoramous 3y agoValid question. I read "increments of £10k" to mean 100 projects/groups or less.
- deleted 3y ago[deleted]
- Hizonner 3y agoWait 'til they find out that the kind of stuff they build is way easier to apply to attack than to defense...
- cookieperson 3y agoYup. Defense is mostly technical nuance. Attacks outside of academia often rely on the weakest links, distracted or scared humans. Chatgpt and LLMs like it can easily mass phish or otherwise thousands of targets in a matter of seconds. Can also be used for OSINT on human targets... Combine the two and ransomware will be a true cottage industry ran by script kiddies. That said there are things they could do to prevent this. They won't do them, it'd be costly research... But they could.
- dinvlad 3y agoWe don’t seem to be quite there yet, but things are moving in that direction, unfortunately (see the Gorilla paper, for example). I still think to tailor the tool to each particular API requires human intelligence (and even that is hard), but a shotgun approach might be more “successful” even if it’s 5% effective. That said, more traditional approaches (non-ML) are probably more effective still.
- gremlinsinc 3y agoWhat about AI security itself, everyone is afraid of rogue AI, but a "dead" AI or "locked up" AI can't do harm. I.e. Have some sort of monitoring system to monitor AI systems, and a sort of kill switch that can disable it from the internet, or if need be trigger a small onboard EMP device to fully nuke the device. It could give some safety if skynet becomes a thing, of course super AI will learn to disable it in a week, but still maybe there's a path there to mitigation of future threats?