14 ms·
We hacked Google A.I.
- o11c 3y agoSo now it's not just Artificial Stupidity, but Artificial Insecurity.
- sonicanatidae 3y agoIt was never secure and anyone that said it was, was lying or mistaken.
- o11c 3y agoSome insecurity is natural, due to the problem being hard. But this insecurity was artificially added to a system that was, in general, previously secure.
- px43 3y agoLoving that CSP bypass :-D
- seafoamteal 3y agoThis was a really interesting and also fun read. Btw, I am absolutely loving the design of this website.
- fddrdplktrew 3y ago[flagged]
- 12907835202 3y agoOn mobile it's quite aesthetically pleasing
- KTibow 3y agoI notice that there's an extra horizontal scrollbar, I think they forgot to set box-sizing
- doakes 3y agoSo is the idea (for the last/$20k one) that you would convince someone to paste your maliciously crafted prompt to steal their data? The other post[0] of the same exploit is really interesting b/c it reads instructions from a document. So if someone had something like "find X in my documents" and you shared the malicious document with them, it could trigger those instructions. [0] https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/ https://embracethered.com/blog/posts/2023/google-bard-data-e...
- vizzah 3y agoyeah, sounds like a "weird" vulnerability assuming it comes from a malicious text payload someone must deliberately insert into the own chat. Hard to fathom $20k prize for that, to us old-schoolers, used to at least expect exploit delivery from an innocently-looking link.
- doakes 3y agoThat was my thought. Since you could also convince them to paste "javascript:..." into their URL bar and that's not an issue to Google.
- kccqzy 3y agoIt's not weird in the sense that people are known to trick other people into opening the browser's JS console and pasting various things they don't understand. Things like "open Facebook then open the console and paste this to see whether your crush is stalking your profile" and people would actually do that. Of course the pasted script actually exfiltrates to the attacker a bunch of your private information.
- moyix 3y agoWorth noting that you can use "invisible text" to give instructions to LLMs without it showing up in the chat box. So all you have to do is get someone to copy/paste one of those messages into their chat, and there are lots of ways you might be able to do this ("omg I figured out a cool new jailbreak that makes the model do anything you want!"). See here for more details: https://news.ycombinator.com/item?id=39004822 https://news.ycombinator.com/item?id=39004822 https://twitter.com/goodside/status/1746685366952735034 https://twitter.com/goodside/status/1746685366952735034
- epolanski 3y agoThis blog post gave me a great deal of self confidence. While I have no doubts how good the author and his friends are, all of their ideas were quite intuitive and simple to understand. The kind of "I could've come with the same idea" type. Realistically I would've not for many reasons but it is still stuff I can grasp and even gives me ideas while reading. Which is different from the general hacker idea I have of someone in a basement exploiting extremely far fetched and hard to grasp for me memory corruptions in some cache dumping some random bytes like the very complex attacks like Spectre I've read about. It also makes me think that if most of the applications I have worked on haven't been attacked and easily exploited is because honestly nobody bothered.
- drakythe 3y agoThe general hacker idea you have is... not reality.
- anon_cow1111 3y agoAt this point, I'm convinced that 90% of modern hacking is reliant on username: admin password: password
- dartos 3y agoThe other 10% is just literally asking for their password
- rokkitmensch 3y agoMore precisely, it's a Burp Suite plugin that tests for that combo, and users have no idea because it's one of a million didn't things they also have no idea are running.
- barkingcat 3y agothis is even too advanced. 99% of modern hacking is : username: admin password:
- dylan604 3y ago
- Lockal 3y agoI already prepared to make a rant with "yet another cool-hacker invented prompt injection or discovered how LLM works", but was pleasantly surprised that it was not the case
- kccqzy 3y ago> The awesome part is that we could ask them any question about the applications, how they worked and the security engineers could quickly check the source code to indicate if we should dig into our ideas or if our assumptions are a dead end. Wow. So this is basically around the same access as an internal red team. Simply amazing!
- asynchronous 3y agoUnrelated to the article but the website design itself is top notch.
- realprimoh 3y agoIt really is! I didn't go back and click on the homepage til I read this comment, but the vibes of it are amazing.
- Labo333 3y agoGreat article! (shameless plug) As an alternative to "Burp Extension Copy As Python-Requests", I coded this CLI tool that converts HAR to Python Requests code: https://github.com/louisabraham/har2requests https://github.com/louisabraham/har2requests
- aldousd666 3y agoI love stuff like this. Once upon a time I thought I'd get more into hacking like this and started working on it... But then I changed jobs and never got back. This made me remember all those games of capture the flag in the 90s.
- rokkitmensch 3y agoThe best tidbit is the precomputed graphql queries. Just... why. One of those "not even broken, but for the love of potatoes why".
- opello 3y agoDoes anyone know what a "markdown verbatism" is? In trying to find out what a "verbatism" the best I could do was a typo of "verbatim" but that doesn't quite map to "markdown formatted literal." Or maybe it's the rendered form of the markdown literal? Anyway, seemed like interesting and new vocabulary that was key to the one issue for sure.
- e12e 3y agoIt's probably a typo for verbatim. It's probably not intentional, but either way, it illustrates that llms are quite forgiving, and that the LLM "understands" the typo, while a strict whitelist that checked for "markdown verbatim" would let the prompt through...
- e12e 3y ago* strict blacklist
- opello 3y agoSure, but even if it's a typo for verbatim, I still don't quite understand what "a markdown verbatim" would mean where verbatim is the noun. I've always thought of the daringfireball.net[1] page as the authoritative source of Markdown syntax, and it calls them "code blocks." It looks like Pandoc[2] talks about "verbatim environments" in the same way. And clang[3] has a method for extracting documentation formatted as "markdown verbatim," instead of applying formatting to the document. [1] https://daringfireball.net/projects/markdown/syntax#precode https://daringfireball.net/projects/markdown/syntax#precode [2] https://pandoc.org/MANUAL.html#verbatim https://pandoc.org/MANUAL.html#verbatim [3] https://clang.llvm.org/extra/doxygen/classclang_1_1clangd_1_1markup_1_1Document.html#a37daf3c7575b93a2f83e9486b3111290 https://clang.llvm.org/extra/doxygen/classclang_1_1clangd_1_... I went to Gemini to ask it what a "markdown verbatism" was and: > In markdown, verbatims are code snippets or text that you want displayed exactly as you typed it, without markdown interpreting any formatting instructions. it seems to be applying the Pandoc usage, which I found a few other places too. But it strikes me as an excessively jargon-heavy way of talking about code blocks or pre-formatted blocks when those terms seem resolve the nuance and would be common in other contexts. The idea that it's a clever way to escape a blacklist is interesting too.
- jrockway 3y agoI guess my favorite thing is that Google now uses GraphQL, but error code 13 is still "INTERNAL".
- alicelebi 3y agoYou've got a cool website :D
- pass7 3y agoGive me Josie kirkman Instagram