10 ms·
We hacked Gemini's Python sandbox and leaked its source code (at least some)
- sneak 2y ago> However, the build pipeline for compiling the sandbox binary included an automated step that adds security proto files to a binary whenever it detects that the binary might need them to enforce internal rules. In this particular case, that step wasn’t necessary, resulting in the unintended inclusion of highly confidential internal protos in the wild ! Protobufs aren't really these super secret hyper-proprietary things they seem to make them out to be in this breathless article.
- daeken 2y agoYeah, this is honestly super interesting as a journey, but not as a destination. The framing takes away from how cool the work really is.
- ratorx 2y agoYup, there’s no reason to believe that the proto files (which are definitions rather than data) are any more confidential than the Gemini source code itself.
- whatevertrevor 2y agoThe protos in question are related to internal authn/z so it's conceivable that having access to that structure would be valuable information to an attacker.
- rurban 2y agoThe protos were already available. See above. A valuable information would be able to run those RPC calls as Principal (their root user)
- film42 2y agoNo, but having the names to the fields, directly from Google, is very helpful for further understanding what's available from within the sandbox.
- kingforaday 2y agoReminds me of this HN article from a month ago with lots of commentary on whether a database scheme is proprietary. https://news.ycombinator.com/item?id=43175628 https://news.ycombinator.com/item?id=43175628
- film42 2y agoYeah there are some interesting similarities. However, the biggest difference is Google has the right to keep source proprietary, and companies like Unity are allowed to provide source code with a reference only license (still proprietary), but the US has FOIA to help push information into the open. Does a DB schema fall under FOIA scope? I think a better question is, can (or is) a db schema being used to conceal information? Is the law attempting to reinforce this barrier? In other words, it should not be about the intent of the requester, but the intent of its owner; and in the case of that article, either by bias in narrative, or the fact that it rhymes with events of the past, there is some tomfoolery about.
- ipsum2 2y agoYes, there's a lot of internal protos from Google that are leaked on the internet. If I recall correctly, it was a hacker News comment that linked to it. Edit: I don't know why the parent comment was flagged. It is entirely accurate.
- kccqzy 2y agoYou are probably thinking of the Google search ranking leak. That leak was the leak of the generated documentation from proto files.
- deleted 2y ago[deleted]
- topsycatt 2y agoThat's the system I work on! Please feel free to ask any questions. All opinions are my own and do not represent those of my employer.
- Mindwipe 2y agoDoes anyone at Google care that you're trying to replace Assistant with this in the next few months and it can't set a timer yet? (I mean it will tell you it's set a timer but it doesn't talk to the native clock app so nothing ever goes off if you navigate away from the window.)
- hnuser123456 2y agoI doubt the guy working on the code sandbox can do anything about the overall resource allocation towards ensuring all legacy assistant features still work as well as they used to. That being said, I was trying to navigate out of an unexpected construction zone and asked google to navigate me home, and it repeatedly tried to open the map on my watch and lock my phone screen. I had to pull over and use my thumbs to start navigation the old fashioned way.
- arebop 2y agoThe Assistant can't reliably set timers either, though I guess 80% is considerably better than 0. Still, I think it used to be better back before Google caught a glimpse of a different squirrel to chase.
- 7bit 2y agoIt can't do shit, especially in some EU countries, where it can do even less shit. Setting timers reminders, calendar events. Nothing. If they kill the assistant, I'll go Apple, no matter how much I hate it.
- GrayShade 2y agoJust tested, you need to enable "Gemini Apps", but they remember your interactions for 3, 18 or 36 months instead of 3 days.
- fpgaminer 2y agoAwww, I was looking forward to seeing some of the leak ;) Oh well. Nice find and breakdown! Somewhat relatedly, it occurred to me recently just how important issues like prompt injection, etc are for LLMs. I've always brushed them off as unimportant to _me_ since I'm most interested in local LLMs. Who cares if a local LLM is weak to prompt injection or other shenanigans? It's my AI to do with as I please. If anything I want them to be, since it makes it easier to jailbreak them. Then Operator and Deep Research came out and it finally made sense to me. When we finally have our own AI Agents running locally doing jobs for us, they're going to encounter random internet content. And the AI Agent obviously needs to read that content, or view the images. And if it's doing that, then it's vulnerable to prompt injection by third party. Which, yeah, duh, stupid me. But ... is also a really fascinating idea to consider. A future where people have personal AIs, and those AIs can get hacked by reading the wrong thing from the wrong backalley of the internet, and suddenly they are taken over by a mind virus of sorts. What a wild future.
- 20after4 2y ago> reading the wrong thing from the wrong backalley of the internet, and suddenly they are taken over by a mind virus of sorts. What a wild future. This already happens to people on the internet.
- tcoff91 2y agoYeah, the way some people lose it from the internet reminds me of Snow Crash.
- paxys 2y agoFunny enough while "We hacked Google's AI" is going to get the clicks, in reality they hacked the one part of Gemini that was NOT the LLM (a sandbox environment meant to run untrusted user-provided code). And "leaked its source code" is straight up click bait.
- dang 2y agoOk, we put the sandbox in the title above. Thanks! (Submitted title was "We hacked Google's A.I Gemini and leaked its source code (at least some part)")
- topsycatt 2y agoThanks!
- infinghxsg 2y agoInstead of sandbox can you just make sure people know it was not a meaningful hack? I mean I “hacked” this site too by those standards.
- HenryBemis 2y agoClick and cash (for the great trio).
- deleted 2y ago[deleted]
- IshKebab 2y ago
- ein0p 2y agoThey hacked the sandbox, and leaked nothing. The article is entertaining though.
- kccqzy 2y agoThey leaked one file in the sandbox that contained lots of internal proto files. The security team reviewed everything in the sandbox and thought nothing in it is sensitive and gave the green light; apparently the review didn't catch this in the sandbox. I guess this is a failing of the security review process, and possibly also how the blaze build system worked so well that people forgot a step existed because it was too automated.
- charcircuit 2y ago>that contained lots of internal proto files So does Google Chrome.
- kccqzy 2y agoNo it's not the same level of internal. There are internal proto files specific to Chromium and its API endpoints, and then there are internal proto files for google3. The latter can divulge secrets about Google's general server side architecture. The former only divulges secrets about server side components relevant to Chromium.
- deleted 2y ago[deleted]
- simonw 2y agoI've been using a similar trick to scrape the visible internal source code of ChatGPT Code Interpreter into a GitHub repository for a while now: https://github.com/simonw/scrape-openai-code-interpreter https://github.com/simonw/scrape-openai-code-interpreter It's mostly useful for tracking what Python packages are available (and what versions): https://github.com/simonw/scrape-openai-code-interpreter/blob/main/packages.txt https://github.com/simonw/scrape-openai-code-interpreter/blo...
- Zopieux 2y agoMeanwhile they could just decide to publish this list in a document somewhere and keep it automatically up to date with their infra. But not, secrecy for the sake of secrecy.
- aleksiy123 2y agoTbh I doubt this is secrecy. More likely just noone has taken the time and effort to do it.
- 12345hn6789 2y agoWhat would the benefit of doing this be?
- simonw 2y agoIt's documentation. Makes it much easier for people to know what kind of problems they can solve using Code Interpreter. It's a bit absurd that the best available documentation for that feature exists in my hacky scraped GitHub repository.
- topsycatt 2y agoThat's a very good point. Let me speak with some folks and see what I can do.
- fudged71 2y agoI just used this package list (and sandbox limitations) to synthesize a taxonomy of capabilities: https://gist.github.com/trbielec/a00a58fa97a232bef8984cc8d0161e5b https://gist.github.com/trbielec/a00a58fa97a232bef8984cc8d01...
- theLiminator 2y agoIt's actually pretty interesting that this shows that Google is quite secure, I feel like most companies would not fare nearly as well.
- kccqzy 2y agoYes and especially the article mentions "With the help of the Google Security Team" so it's quite collaborative and not exactly black box hacking.
- jll29 2y agoRunning the built-in "strings" command to extract a few file names from a binary is hardly hacking/cracking. Ironically, though, getting the source code of Gemini perhaps wouln't be valuable at all; but if you had found/obtained access to the corpus that the model was pre-trained with, that would have been kind of interesting (many folks have many questions about that...).
- dvt 2y ago> but if you had found/obtained access to the corpus that the model was pre-trained with, that would have been kind of interesting Definitionally, that input gets compressed into the weights. Pretty sure there's a proof somewhere that shows LLM training is basically a one-way (lossy) compression, so there's no way to go back afaik?
- jdiff 2y agoNot the original, but a lossy facsimile that's Good Enough for almost anything. And as the short history of LLMs and other nets has shown us, they're often not even all that lossy.
- tgtweak 2y agoThe definition of hacking is getting pretty loose. This looks like the sandbox is doing exactly what it's supposed to do and nothing sensitive was exfiltrated...
- jeffbee 2y agoI guess these guys didn't notice that all of these proto descriptors, and many others, were leaked on github 7 years ago. https://github.com/ezequielpereira/GAE-RCE/tree/master/protos https://github.com/ezequielpereira/GAE-RCE/tree/master/proto...
- bluelightning2k 2y agoCool write up. Although it's not exactly a huge vulnerability. I guess it says a lot about how security conscious Google is that they consider this to be significant. (You did mention that you knew the company's specific policy considered this highly confidential so it does count but it feels a little more like "technically considered a vulnerability" rather than clearly one.)
- parliament32 2y ago> resulting in the unintended inclusion of highly confidential internal protos in the wild I don't think they're all that confidential if they're all on github: https://github.com/ezequielpereira/GAE-RCE/tree/master/protos/security https://github.com/ezequielpereira/GAE-RCE/tree/master/proto...
- saagarjha 2y agoI mean, those were also disclosed via a vulnerability.
- Brian_K_White 2y agoBut it still means they aren't guilty of leaking/disclosing them. It's not a valid point of criticism. The escape did not in fact "result" in the leak of confidential photos. That already happened somewhere else. This only resulted in the republishing of something already public. Or another way, it's not merely that they were already public elsewhere, the imortant point is that the photos were not given to the ai in confidence, and so re-publishing them did not violate a confidence, any more than say github did. I'm no ai apologist btw. I say all of these ais are committing mass copyright violation a million times a second all day every day since years ago now.
- saagarjha 2y agoI’m not criticizing them
- Brian_K_White 2y agoThe article made that criticism.
- saagarjha 2y agoThe article criticized its authors? I’m not sure I understand.
- curiousZeedX 2y ago[dead]
- qwertox 2y agoSuper interesting article. > but those files are internal categories Google uses to classify user data. I really want to know what kind of classification this is. Could you at least give one example? Like "Has autism" or more like "Is user's phone number"?
- StephenAmar 2y agoThe latter. Like is it a public ID, an IP, user input, ssn, phone number, lat/long… Very useful for any scenario where you output the proto, like logs, etc…
- commandersaki 2y agoTheir "LLM bugSWAT" events, held in vibrant locales like Las Vegas, are a testament to their commitment to proactive security red teaming. I don't understand why security conferences are attracted to Vegas. In my opinion its a pretty gross place to conduct any conference.
- numbsafari 2y agoYou answered your own question.
- zem 2y agorelatively cheap event space and hotels. it's hard to find a city to host a large conference.
- hashstring 2y agoReal, I feel the exact same way.
- desmosxxx 2y agoWhat don't you understand. Vegas is literally built for conferences.
- scudsworth 2y agoreinvent is in vegas
- lmm 2y agoExcluding uptight scolds is a feature not a bug. There's a lot of overlap between people who find Vegas objectionable and people who find red teaming objectionable (because why would any decent person know attacking/exploiting techniques).
- commandersaki 2y agoThe irony is that Vegas takes a dim view of those that take advantage of their gaming venues. The institutions that run it are quite aggressive when it comes to being attacked. Anyways, security conferences such as BSides run all over the world in various cities where red teaming type activities is embraced. IMO it'd be nice to diversify from Vegas, preferably places with more scenery/greenery like Boulder or something.
- b0ner_t0ner 2y agoVery distracting background/design on desktop; had to toggle reader view.
- lqstuart 2y agoSo by “we hacked Gemini and leaked its source code” you really mean “we played with Gemini with the help of Google’s security team and didn’t leak anything”
- worldsavior 2y agoSad that I didn't read this comment before reading this article.
- Cymatickot 2y agoProbably best text I've seen in AI train ride recently: """"" As companies rush to deploy AI assistants, classifiers, and a myriad of other LLM-powered tools, a critical question remains: are we building securely ? As we highlighted last year, the rapid adoption sometimes feels like we forgot the fundamental security principles, opening the door to novel and familiar vulnerabilities alike. """" There this case and there many other cases. I worry for copy & paste dev.
- mr_00ff00 2y agoSlightly irrelevant, but love the color theme on the python code snippets. Wish I knew what it was.