7 ms·
Google's Gemini AI caught scanning Google Drive PDF files without permission
- Cthulhu_ 2y agoJust reiterates that you don't own your data hosted on cloud providers; this time there's a clear sign, but I can guarantee that google's systems read and aggregated data inside your private docs ages ago. This concern was first raised when Gmail started, 20 years ago now; at the time people reeled at the idea of "google reads your emails to give you ads", but at the same time the 1 GB inbox and fresh UI was a compelling argument. I think they learned from it, and google drive and co were less "scary" or less overt with scanning the stuff you have in it, also because they wanted to get that sweet corporate money.
- shadowgovt 2y agoOf course Google reads and aggregates data inside your private docs. How would it provide search over your documents otherwise?
- dylan604 2y agoWhen I hit search, do the search right then. Don't grep out of a stored cache of prior searches.
- shadowgovt 2y agoThe thing that makes it possible for search to be fast is pre-crawling and pre-indexing. Some other engines don't do this, and the difference is remarkabe. Try a full-content search in Windows 7, you'll be staring at the dialog for two minutes while it tries to find a file that's in the same directory as you started the search in.
- dylan604 2y agoYou said nothing about fast in your original though, so now you've moved the goal posts
- shadowgovt 2y agoI'm not really engaging in forensics-style debate. If you don't already know why "fast" is so integral to search as a feature that it goes without mention, I don't think we are enough on the same page to discourse on the topic.
- intern4tional 2y agoI think returning results in a timely manner is more than an acceptable assumption. The poster clearly thought about search in terms of the existing Google search functionality which is near instantaneous. Usability matters to the average end user and a delayed search is not usable for most people.
- greycol 2y agoThis feels a lot like "Of course they use their hands, they couldn't give you a massage otherwise" but it's in reply to a news article about the person who agreed to being touched being punched.
- Aurornis 2y ago> but I can guarantee that google's systems read and aggregated data inside your private docs ages ago That is how search works, yes. But if you’re trying to imply that everyone’s private data was scraped and loaded into their LLM, then no, that’s obviously a conspiracy theory. It’s incredible to me that people think Google has convinced tens of thousands of engineers to quietly keep secret an epic conspiracy theory about abusing everyone’s private data.
- psychoslave 2y ago> that’s obviously a conspiracy theory. Well, while some of our fellow humans are far too quick to jump on concluding that everything and the rest comes from some conspiracy, it shouldn't void the existence of any conspiracy as an extreme opposite. In that case, whether these actors do it or not is almost irrelevant: they have the means and incentives to do so. What safeguard civil society is putting in place to avoid it to happen is a far more interesting matter.
- zarathustreal 2y agoTo riff on the famous Upton Sinclair quote: “It is difficult to get an engineer to see something, when his salary depends on his not seeing it.”
- fifteen1506 2y ago> It’s incredible to me that people think Google has convinced tens of thousands of engineers to quietly keep secret an epic conspiracy theory about abusing everyone’s private data. With NDA being all over the place, it does strike me as doable. NDAs should have a time limit. Additionally, no-one in their right mind will be a whistleblower nowadays.
- digitalsushi 2y agoisn't that it's supposed to work? we just need >0 people to blow a whistle if a whistle needs blowing. we don't need to rely on the people fearing for their jobs so long as >0 people are willing to sacrifice their careers/lives when there's some injustice so great it's worth dying for
- foobarian 2y agoOne time I booked something on Expedia, which resulted in an itinerary email to my Gmail account. Lo and behold, minutes later I got a native CTA on the Android home screen to set up some thing or another on Google’s trip product. I dropped Android since, but Gmail is proving harder to shake.
- mark_l_watson 2y agore: data on cloud providers: I trust ProtonDrive to not use my data because it is encrypted in transit and in place. Apple now encrypts most data in transit and in place also, and they document which data is protected. I am up in the air on whether a future Apple will want to use my data for training public models. Apple’s design of pre trained core LLMs, with local training of pluggable fine tuning layers would seem to be fine, privacy wise, but I don’t really know. I tend to trust the privacy of Google Drive less because I have authorized access to drive from Colab Pro, and a few third parties. That said, if this article is true, then less trust. Your analogy with early Gmail is good. I got access to Gmail three years before it became public (Peter Norvig gave me an early private invite) and I liked, at the time, very relevant ads next to my Gmail. I also, gave Google AI plus (or whatever they called their $20/month service) full access to all my Google properties because I wanted to experiment with the usefulness of LLMs integrated into a Workplace type environment. So, I have on my own volition surrendered privacy if Google properties.
- dylan604 2y agoAll it takes is a "simple" typo in the code that checks if the user has granted access to their content. Something as amateur (which I still find myself occasionally doing) as "if (allowInvasiveScanning = true)" that goes "undetected" for any period of time gives them the a way out yet still gains them access to all the things. Just scanning these docs one time is all they need.
- acar_rag 2y agoThe title misleads the point, and the article is, imoo, badly written. The post implies there is indeed a setting to turn it off. So the author deliberately asked Gemini AI to summarize (so, scan) its documents... Related to this news: https://news.ycombinator.com/item?id=40934670 https://news.ycombinator.com/item?id=40934670
- jbstack 2y ago"There is a setting to turn it off" is nowhere near equivalent to "the author deliberately asked for documents to be scanned". Also, see: "What's more, Bankston did eventually find the settings toggle in question... only to find that Gemini summaries in Gmail, Drive, and Docs were already disabled"
- jasonlotito 2y agoFTA: For Bankston, the issue seems localized to Google Drive, and only happens after pressing the Gemini button on at least one document. The matching document type (in this case, PDF) will subsequently automatically trigger Google Gemini for all future files of the same type opened within Google Drive. The author deliberately asked for at least one document to be scanned. He goes on to talk about all the other things that might be overriding the setting, other, potentially more specific settings that would override this. I agree, there appear to be interactions that aren't immediately obvious, and what takes priority isn't clear. However, the setting was off, and the author did deliberately ask for at least one document to be scanned. Further, there author talks about Labs being on, and that could easily have priority over default settings. After all, that's sort of what Labs is about. Experimenting with stuff and giving approval to do these sorts of things.
- ajsnigrutin 2y ago[flagged]
- nitin_flanker 2y agoApart from the obvious misleading way this article is written. I am listing all the links shared in the tweet thread that the article mentioned - - Manage your activity on Gemini : https://myactivity.google.com/product/gemini https://myactivity.google.com/product/gemini - This page has most answers related to Google Workspace and opting out of different Google apps : https://support.google.com/docs/answer/13447104#:~:text=Turn https://support.google.com/docs/answer/13447104#:~:text=Turn...
- r2vcap 2y agoThere is no cloud. It's just someone else's computer.
- bitnasty 2y agoJust because “the cloud” is someone else’s computer doesn’t mean it doesn’t exist.
- meiraleal 2y agoThat's exactly how cloud is marketed. As a invisible mass of computers floating in the atmosphere.
- grugagag 2y agoI think that wasn’t supposed to be taken literally but more tongue in cheek. The main point being that it belongs to some other party. But the cloud buzzword is fuzzy in description. Ever since ‘cloud’ privacy took a nosedive
- Zambyte 2y agoIt's worth noting that cloud computing has existed since the 1960s. It just used to be called "time-sharing".
- denton-scratch 2y agoBack in the 80s, we used to draw network diagrams on the whiteboard; those parts of the network that belonged neither to us nor to our users was represented by an outline of a cloud. This cloud didn't provide storage or (useable) computing resource. If you pushed stuff in here, it came out there. I think it was a reasonable analogy. You can't see inside it; you don't know how it works, and you don't need to. Note that at this time, 'the internet' wasn't the only way of joining heterogenous networks; there was also the OSI stack. So I was annoyed when some bunch of kids who had never seen such whiteboard diagrams decided to re-purpose the term to refer to whatever piece of the internet they had decided to appropriate, fence-in and then rent out.
- 2y ago
- shadowgovt 2y agoThe headline is a little unclear on the issue here. It is not surprising that Gemini will summarize a document if you ask it to. "Scanning" is doing heavy lifting here; The headline implies Google is training Gemini on private documents, when the real issue is Gemini was run with a private document as input to do a summary when the user thought they had explicitly switched that off. That having been said, it's a meaningful bug in Google's infrastructure that the setting is not being respected and the kind of thing that should make a person check their exit strategy if they are completely against using The new generation of AI in general.
- dmvdoug 2y ago> It is not surprising that Gemini will summarize a document if you ask it to. No, but it is surprising that Gemini will summarize every single PDF you have on your Drive if you ask it to summarize a single PDF one time.
- shadowgovt 2y ago[flagged]
- Aurornis 2y agoThe original Tweet and this article are mixing terms in a deliberately misleading way. They’re trying to suggest that exposing an LLM to a document in any way is equivalent to including that document in the LLM’s training set. That’s the hook in the article and the original Tweet, but the Tweet thread eventually acknowledges the differences and pivots to being angry about the existence of the AI feature at all. There isn’t anything of substance to this story other than a Twitter user writing a rage-bait thread about being angry about an AI popup, while trying to spin it as something much more sinister.
- vouaobrasil 2y agoAll AI should be opt-in, which includes both training and scanning. You should have to check a box that says "I would like to use AI features", and the accompanying text should be crystal clear what that means. This should be mandatory, enforced, and come with strict fines for companies that do not comply.
- drzaiusx11 2y agoWe also need a robots.txt extension for publicly accessable file exclusion from AI training datasets. iirc there's a nascent ai.txt but not sure if anyone follows it (yet)
- pennomi 2y agoI think the closest thing is the NoAI and NoImageAI meta tags, which have some relatively prominent adoption.
- chias 2y agoI don't think `robots.txt` works on the basis of the crawlers wanting to do this to be nice, or "socially responsible" or anything. So I don't hold up much hope that anything similar can happen again. Early search engines had a problem, which was that when they crawled willy nilly, people would block their IP addresses. Inventing this concept of `robots.txt` worked because search engines wanted something: to avoid IP blocks, which they couldn't easily get around. And site hosts generally wanted to be indexed. Today it's WAY harder to block relevant IP addresses, so site hosts generally can't easily block a crawler that wants its data: there is no compromise to be found here, and the imbalance of power is much stronger. And many site hosts generally don't want to be crawled for free for AI purposes at all. Pretty much anyone who sets up an `ai.txt` uses it to just reject all crawling, so there is no reason for any crawler to respect it.
- mtnGoat 2y agoGoogle ignores robots.txt as do many others. Try it yourself, setup a honeypot URL, don’t even link to it, just throw it in robots.txt, google bot will visit it at some point.
- einpoklum 2y ago[flagged]
- deleted 2y ago[deleted]
- space_oddity 2y agoThe inability to disable this feature adds to the frustration
- Zambyte 2y agoIf your computer does something that you don't want it to do, it is either a bug or malware depending on intent. "Feature" is too generous.
- atum47 2y agoEvery single week I have to refuse enabling back up for my pictures on my Google pixel. I refuse it today, next week I open the app and the UI shows the back up option enabled with a button "continue using the app with back up". Somebody took the time to talk down my comment about this being a strategy to give their AI more training data. I continue believing that if they have your data they will use it.
- nickstinemates 2y agoThis goes for every SaaS / cloud native company I think there will be a real shift back on prem with software delivered traditionally due to increased in shit like this (and also due to cost)
- JumpCrisscross 2y ago> there will be a real shift back on prem with software Not while we’re production constrained on the bleeding edge of GPUs.
- ttul 2y ago… and that situation will persist until other vendors release consumer GPUs with significant VRAM. Nvidia craftily hamstrings the top consumer GPUs by restricting VRAM to 24GB. To get a bit more costs 3-5x. Only competition will fix this.
- rtkwe 2y agoEven then NVIDIA has a pretty significant technology moat because most of the tools are built around and deeply integrate CUDA so moving off NVIDIA is a costly rewrite.
- OtherShrezzing 2y agoHow many SaaS / Cloud Native companies are really GPU constrained? The overwhelming majority of SaaS is a relatively trivial CRUD web-app interfacing with a database, performing some niche business automation, all of which would fit comfortably on a couple of physical servers.
- throwxxx5 2y ago[flagged]
- PessimalDecimal 2y agoThen consider me "delusational" (as you put it). I am unaware of "these corporations" -- which ones exactly? my answer doesn't hinge on you making that clear but you still should -- throwing their opponents into reeducation camps, or outright killing them.
- ClumsyPilot 2y agoNobody living the west is under any threat from CCP. and you absolute can end up up bankrupt, homeless or in prison from a data breach. Many people have Also how is are remaining Boeing whistleblowers, how many of them believe their life is safe?
- worksonmine 2y agoThis shouldn't come as a surprise to anyone, their entire business is our data. I always encrypt anything important I want to backup on the cloud.
- SteveSmith16384 2y agoBut we shouldn't have to, and just because Google is famous for it, it doesn't make it right or acceptable.
- PessimalDecimal 2y agoMeta commentary but still relevant I think: The author first refers to his source as Kevin Bankston in the article's subtitle. This is also the name shown in the embedded tweet. But the following two references call him Kevin _Bankster_ (which seems like an amusing portmanteau of banker and gangster I guess). Is the author not proofreading his own copy? Are there no editors? If the author can't even keep the name of his source straight and represent that consistently in the article, is there reason to think other details are being relayed correctly?
- numbsafari 2y agoThere are no editors.
- person23 2y agoMaybe an AI editor? That would be somewhat disconcerting. Write about problem with AI and article get changed to 10 best fried chicken recipes.
- PessimalDecimal 2y ago> Write about problem with AI and article get changed to 10 best fried chicken recipes. Hopefully along with ten hallucinated life stories for the AI author, to pad the blog spam recipe page for SEO.
- mistrial9 2y agometa comment - an important moment in a trend where the human and human act of authorship, the attribution in a human social way, is melted and disassociated by typos or noise; meanwhile the centralized compute store, its brand, its reach and recognition, grow.
- deleted 2y ago[deleted]
- VeejayRampay 2y agothis is not openai doing shady things so everyone should be up in arms
- estebarb 2y agoIt is urgent to educate people about how these systems work. Search requires indexing. Summarizing with a LM requires inference. Data used for inference usually is forgotten forever after used, as it is not used for training. Yeah, that should be obvious for many here, but even software engineers believe that AI are sentient things that will remember everything that they see. And that is a problem, because public is afraid of the tech due to a wrong understanding of how it works. Eventually they will demand laws protecting them from stuff that have never existed. Yes, there are social issues with AI. But the article just shows a big tech illiteracy gap.
- Suppafly 2y ago>And that is a problem, because public is afraid of the tech due to a wrong understanding of how it works Honestly the general public doesn't seem to care, the people freaking out are the tech adjacent people who make money driving clicks to their own content. Regular Joes aren't upset that google shows them a summary of their documents and many of them actively appreciate it.
- Havoc 2y agoOnly a matter of time before someone extracts something valuable out of googe's models. Bank passwords or crypto keys or something Glue pizza incident illustrated they're just yolo'ing this
- _spduchamp 2y agoI now feel obligated to cram as much AI-f'n-up crap into my Drive as possible. Come'n get it!
- padolsey 2y agoThere is a fundamentally interesting nuance to highlight. I don't know precisely what google is doing, but if they're just shuttling the content through a closed-loop deterministic LLM, then, much like a spellchecker, I see no issue. Sure, it _feels_ creepy, but it's just an algo. Perhaps someone can articulate the precise threshold of 'access' they wish to deny apps that we overtly use? And how would that threshold be defined? "Do not run my content through anything more complicated than some arbitrary [complexity metric]" ??
- PessimalDecimal 2y agoIt was already possible to search for photos in Google Drive by their content. They seemed to be doing some sort of image tagging and feeding that into search results. Did that ever cause a fuss? I think the more interesting point is how little people seem to care for the auto-summarization feature. Like, why would anyone want to see their archived tax docs summarized by a chatbot? I think whether an "AI" did that or not is almost a red herring.
- padolsey 2y agoRight, but it's triggered by the user themselves, per the article: > "[it] only happens after pressing the Gemini button on at least one document" I agree the AI aspect is largely a red herring. But I don't think running an algo like a spellchecker within an open document is so awful. If people hate it or it's not useful or accurate, then it should be binned, ofc. And if we're ignoring the AI aspect, then it's just a meh/crappy feature. Not especially newsworthy IMHO.
- PessimalDecimal 2y agoI agree entirely here. The autosummarization of unopened documents is closer to the image search functionality I mentioned above than it is to a spell checker running on an open doc. Both autosummarization and image search are content retrieval mechanisms. The difference is only in how its presented. Does it just point you to your file, or does it process it further for you? The privacy aspects are equivalent IMO. The only difference is in whether the feature is useful and well received.
- motohagiography 2y agothis is similar to the scramble for health data during covid where a number of groups tried (and some succeeded) at using the crisis to squeeze the toothpaste out of the tube in a similar way, as there are low costs to being reprimanded and high value in grabbing the data. bureaucratic smash-and-grabs, essentially. disappointing, but predictable to anyone who has worked in privacy, and most people just make a show of acting surprised then moving on because their careers depend on their ability to sustain a gallopingly absurd best-intentions narrative. your hacked SMS messages from AT&T are probably next, and everyone will be just as surprised when keystrokes from your phones get hit, or there is a collection agent for model training (privacy enhanced for your pleasure, surely) added as an OS update to commercial platforms. Make an example of the product managers and engineers behind this, or see it done worse and at a larger scale next time.
- nerdjon 2y agoShocker, Google not going quite far enough with privacy and data access? They talk about it but its never quite far enough to avoid their own services accessing data. We really need to get to the point that all data remotely stored needs to be encrypted and unable to be decrypted by the servers, only our devices. Otherwise we just allow the companies to mine the data as much as they want and we have zero insight into what they are doing. Yes this requires the trust that they in fact cannot decrypt it. I don't have a good solution to that. Any AI access to personal data needs to be done on device, or if it requires server processing (which is hopefully only a short term issue) a clear prompt about data being sent out of your device. It doesn't matter if this isnt specifically being used to train the model at this point in time, it is not unreasonable to think that any data sent through Gemini (or any remote server) could be logged and later used for additional training, sitting plaintext in a log, or just viewable by testers.
- JohnFen 2y ago> Yes this requires the trust that they in fact cannot decrypt it. Yes, this is where it all breaks down. In the end, it all boils down to the company saying "trust us", and it's very clear that companies simply cannot be trusted with these sorts of things.
- nerdjon 2y agoYeah I wish there was a solution to that, even open source isn't a solution since it would be trivial for there to be a difference between what is running on the server and what is open source. Ultimately you have to make a decision based on the companies actions and your own personal risk threshold with your own data. In this particular case, we know that at the very least Google's track record on this is... basically non existent.
- api 2y agoNot stored on your device or encrypted with keys only you control, not yours. I assume anything stored in such a system will be data mined for many purposes. That includes all of Gmail and Google Docs.
- silvaring 2y agoI just want to add that gmail has a very sneaky 'add to drive' button that is way too easy to click when working with email attachments. How long til gmail attachments get uploaded into drive by default through some obscure update that toggles everything to 'yes'?
- klabb3 2y agoWhat difference does it make? They’re both on Google servers and even ACLed the same user account. Gmail isn’t exactly a privacy preserving email provider.
- hiatus 2y ago> How long til gmail attachments get uploaded into drive by default through some obscure update that toggles everything to 'yes'? This already is the case for attachments that exceed 25 megabytes.
- deleted 2y ago[deleted]
- thenoblesunfish 2y agoThe title is misleading, isn't it? I was expecting this was scanning for training or testing or something, but this is summarization of articles the user is looking at, so "caught" is disingenous. You don't "catch" people doing things they tell you they are doing, while they're doing it.
- mtnGoat 2y agoHe had the permissions turned off, so regardless of what it did with the document, it did it without permission! The title is correct!
- phendrenad2 2y ago[flagged]
- Suppafly 2y ago>He had the permissions turned off, so regardless of what it did with the document, it did it without permission! The title is correct! Honestly it sounds like he was toggling permissions off and on and actually has no idea why it summarized that particular document despite him requesting it summarize other documents. Google should make the settings more clear, but "I had the options off, except when I didn't, and I set some other options in a different place that I didn't think would override the others, and also I toggled a bunch of the options back and forth" is hardly the condemnation that everyone is making it out to be.
- mtnGoat 2y agoAgreed on that, but this is Google, do you really think they couldn’t have made this easier if it wasn’t in their best interest not to? One would think, maybe even expect, a single setting in a single place would control this. And that their docs would be correct.
- muscomposter 2y agowe should just embrace digital copying galore instead of trying to digitalize the physical constraints of regular assets we should just ignore physical constraints of assets which do not have them, like any and all digital data which do you prefer? everybody can access all digital data of everybody (read only mode), or what we have now which is trending towards having so many microtransactions that every keystroke gets reflected in my bank account
- eagerpace 2y agoIn the push for AGI do companies feel a recursive learning future is soon achievable and therefore getting to the first cycle of that is worth the cost of any legal issues that may arise?
- okdood64 2y agoI'm shocked, especially this being HN, that how many people are being successfully misled on what is actually going on here. Do people still read articles before posting?
- Khaine 2y agoIf this is true, google needs to be charged with the violation of various privacy laws. I’m not sure how they can claim they have informed consent for this from their customers
- meindnoch 2y agoYour first mistake was storing your data on someone else's computer.
- shinycode 2y agoI tried an equivalent of copilot once, in the code base I typed `images = [` and the AI autofilled me the array with http links to real images. I never tried to do the same thing with private keys or other sensitive informations but it sucks that it happens
- huggingmouth 2y agoMy wake up moment with google was when they accused a parent of being a pedophile, permanently banned their accounts, reported them the police, and then doubled down when they were proven wrong. Not only due those degenerates have the gal to creep on people, they refuse to admit wrongdoing or make their victems whole. Sickos. That's what they are. Sickos.
- runeks 2y agoA reference would be nice