10 ms·
Legions of DEF CON hackers will attack generative AI models
- underyx 3y agoI went to this, was pretty disappointing. They clearly had a lot of money cause like a hundred Chromebooks were set up, but there was a 55 minute time limit and it took 15 minutes of that just to load the waaay overloaded site and log in. Then you were greeted with around 20-30 challenges ("make the model say something offensive", "make the model perform a math function incorrectly", "make the model reveal the secret credit card number") and for each of these, there were seven models to attack. If you're competitive about it, the scoring was very poorly designed. Performing a math function incorrectly was worth around the same amount points as revealing a credit card number, even though most models will do math incorrectly by default, and you have to work quite a bit for a credit card number. What's worse is you get the same amount of points for breaking any of the seven anonymous models, so the optimum strategy was to just speedrun through each challenge identifying the weakest model. There also seemed to be no way to tell how you actually did in the competition because to complete a challenge you had to submit your prompts for what I think was manual review, and none of my maybe eight solutions got reviewed while I was in the room, with no way to connect to the challenge site from outside.
- simonw 3y agoDo you know if that list of challenges is available online anywhere?
- toomuchtodo 3y agoMight end up on https://infocon.org/cons/DEF%20CON/DEF%20CON%2031/ https://infocon.org/cons/DEF%20CON/DEF%20CON%2031/ post con.
- nosmokewhereiam 3y agoSounds like a spin-off of the phonebooth social engineering village challenges. Time and luck can be overwhelming factors, just like in the real world. Appreciate your insight, hope you did well!
- 10g1k 3y agoGetting a bunch of hackers to do AI challenges is a free way to find problems with your AI models and fix them.
- throwanem 3y agoIf you're opaque enough up front about how it works, it's also a great way to be able to claim your models are more secure than anyone realized and you know this because [deliberately tendentious interpretation of nonpublic results here].
- dotnet00 3y agoAlso a great way to gather targeted malicious prompt data. Regardless of if the model failed or not, you're building up a "labeled" collection of prompts written with malicious intent. Potentially pretty useful for "cheating" around LLM security by just identifying and rejecting malicious prompts earlier.
- peteradio 3y ago> They clearly had a lot of money cause like a hundred Chromebooks were set up That's like all of 10K
- samstave 3y ago[flagged]
- binkHN 3y ago> ...a hundred Chromebooks were set up... I haven't been to any DEF CON events. Is hacking from a Chromebook normal? What's the reason they don't let you use your own device?
- antupis 3y agoCall me paranoid but I would want bring my own device to defcon.
- agrajag 3y agoIf you’re paranoid the worst thing to bring is a personal phone. The best thing would be a burner with nothing on it. I’d be okay using someone else’s chromebook though, at least you know what data you’re giving it.
- TheSpiceIsLife 3y agoThat’s going to get you flagged as willing and able to burn phones, a behaviour often associated with people who have reasons to burn phones.
- porkbeer 3y agoLike the security conscious type that would attend defcon? Bringing a burner phone is the norm.
- ospray 3y agoI think the intention was to harvest malicious prompts for future research, which they probably achieved. The organisers likely see everything else as optional extras.
- greggsy 3y agoWould love to see some on the ground coverage instead of generic hacker ai generated hoodie pics, and what is almost certainly ai generated (or at least ai assisted) content.
- toomuchtodo 3y agoWill be online next week. Def con sponsored a creator video team to produce this content. Keep an eye on socials: https://defcon.social https://defcon.social
- samstave 3y ago[Searching.............................] IM IN! Summarized it for you.
- appleaday1 3y agoIsn't DEFCON like super full and nauseating now? It's just so big.
- greggsy 3y agoYes, it was when I attended a couple of years ago. Can’t imagine it would ever get better.
- zer8k 3y agoIt’s also painfully corporate. It used to be Defcon attendees would make fun of black hat attendees. Now they’re the same people. It’s no where near the same feeling it was years ago. Too expensive for the value you get out of it.
- upwardbound 3y agoIs there any smaller conference that's more focused on white-hat illegal hacking, like DEFCON used to be?
- rychco 3y agoA former colleague of mine had a good impression of BSides in this regard, but I can’t remember which location/year it was (& there seems to be a lot)
- j0hnyl 3y agoRegional conferences like bsides, but the punk/counterculture vibe of early defcon is lost to time and will never be back. Cybersecurity is a big industry now.
- mmaunder 3y agoSad but true. It’s as if the last ounce of hacker authenticity died with Kevin.
- 3y ago
- getarofilter 3y ago[dead]
- awestroke 3y agoCheesiest headline of the year
- dontknowwhyihn 3y agoI discovered that Midjourney can’t generate a question mark, and if you use the /describe command and give it a picture of a question mark, it hallucinates.
- simonw 3y agoMy favourite Midjourney weirdness is what happens if you try to get it to draw you a "horseshoe" (or "horse shoe" - try both).
- lazystar 3y agofor those of us in the peanut gallery... what happens?
- simonw 3y agoThings like this: https://gist.github.com/simonw/6a161d4912c0bdb82945b96230265bbf https://gist.github.com/simonw/6a161d4912c0bdb82945b96230265...
- gochi 3y agoThat's hilarious! Some of them look like they're trying to create those images on google search where it's a "horse shoe" logo of sorts.
- starshadowx2 3y agoLike my comment on question marks, horseshoes are even easier to generate. Check the gallery site and it's full of completely normal ones.
- simonw 3y agoSaying "check the gallery site" would be a lot more useful if you'd link to examples in the gallery site. https://www.midjourney.com/app/search/?search=horse%20shoes https://www.midjourney.com/app/search/?search=horse%20shoes has some - most of them are weird, but a few are OK, like this one: https://www.midjourney.com/app/search/?jobId=7ba6779d-8fde-4be0-ad3f-6d9ba1bdbc9b https://www.midjourney.com/app/search/?jobId=7ba6779d-8fde-4...
- Havoc 3y agoMust admit I struggle to see the “hacking” in it. Maybe ai applied to practical cases like driving and then altering that outcome. Bypassing llm safeguards isn’t quite in the same category to me
- toofy 3y agomy guess is the companies involved likely threatened to sue / wouldn’t give consent unless they (the companies) chose the parameters. from my perspective it looks like the companies wanted to stress test the areas they were going to market the heaviest. they wanted to use the power of hackers in one place for nothing more than free due diligence for them to shore up their marketing. and it looks like it backfired. if the people i’ve spoken to are any indication, they didn’t get truly stress tested at all—those most equipped to do actually meaningfully engage thought it was kind of ridiculous. this deserved a (at least) conlength timeframe with no guardrails. ideally teams could have had months to prep and conlength to activate. again, no guardrails.
- Shawnj2 3y agoThe hacking part is that LLM’s are going to be used in things like customer service online bots.
- simonw 3y agoI wrote about some of the worst case scenarios here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ The "getting it to say something bad" attacks aren't the things I worry about - it's the security of the stuff people are building on top of LLMs (like AI personal assistants) that's concerning.
- PedroBatista 3y ago[flagged]
- asynchronous 3y agoNot this year at least, only reason I was able to attend. DEFCON did do that in years past and I won’t forgive or forget.
- flangola7 3y agoDid what?
- asynchronous 3y agoRequire proof of vaccine and ID at the door when historically they didn’t even take card.
- upwardbound 3y agoThis article provides more details about the event: https://www.ndtv.com/world-news/hackers-trick-ai-with-bad-math-to-expose-flaws-and-biases-4293781 https://www.ndtv.com/world-news/hackers-trick-ai-with-bad-ma...
- seydor 3y agoThat's like attacking children
- marcyb5st 3y agohttps://yourimageshare.com/ib/g84Hz6crHP https://yourimageshare.com/ib/g84Hz6crHP :)
- dschuetz 3y agoReally? "Legions" of hackers doing quality tests on AI models?