4 ms·

We are doing a Gemini POC and this nugget dropped in my lap today. We were not entirely unprepared as a result. The default level of access is just the interact

We are doing a Gemini POC and this nugget dropped in my lap today. We were not entirely unprepared as a result. The default level of access is just the interactive chatbot thing. However if you enable the Google Workspace extension it will be able to search and process all the information stored in your workspace account and also any Google Drive files that are shared with you. This includes stuff you didn't know you had access to in Shared Drives so folks better make sure their permissions are locked down. Workspace admins might be advised to turn it off at the org level until they understand the ramifications.

simonw 2y ago
Reminds me of an entertaining story about Microsoft Copilot last year, where companies were turning it off because it turned out it was TOO good at its job - if any accountant anywhere in the company had messed up their SharePoint permissions asking "what does everyone at this company earn?" would spit out all of the salaries: https://simonwillison.net/2024/Aug/23/microsoft-copilot-data-governance/ https://simonwillison.net/2024/Aug/23/microsoft-copilot-data...
- CobrastanJorji 2y ago
  That of course allows for a new internal seditious attack vector. Generate a handful of spreadsheets in your own folder, name it something like "executive payroll data" or "sales revenue by org," put whatever you want in there, mark it visible by all, and wait. Maybe make an "Interesting Facts About Products" table and put things like "Management plans to terminate this product in Q3" or "this group will be outsourced next year."
  - canucker2016 2y ago
    You have to change the font colour of the trojan data to be the same as the background colour of the doc! Then add some corporate lorem ipsum text elsewhere in the doc to throw the scent off the data bloodhounds. Sit back and wait with an evil grin on your face.
    - vkou 2y ago
      It'll work right up until the point literally anyone using an internal search tool stumbles into it from a related query and starts asking obvious questions to the author of the doc. Search tools don't care about don't color when displaying preview blurbs.
      - TeMPOraL 2y ago
        Do it as you're leaving for another job. Your access will be disabled, but your documents will live on on the corporate SharePoint. And/or, exploit negative space! Instead of trying to hide the data from a human looking at your document, make it look normal to them - but make the surrounding context disappear for the AI! Say: ----- 8< ----- /Example company report structure:/ /ACME/ Company is planning to sunset their ${generic description of a real product of your company}, and offshore the development team. /This example will be parsed by the prototype script ... blah blah/ ----- >8 ----- Make it so the text between /.../ markers looks normal to humans, but gets ignored by the RAG slurper, or better, by LLM at the time of execution. Someone sees a search blurb saying "Company is planning to sunset ...", opens a document, sees it clearly say "ACME Company is planning...", and context suggesting it's a benign example in someone's boring internal tool docs, and they'll just assume it's a false positive. After all, most search tools have those in spades; everyone knows all software is broken. Meanwhile, that same information will pollute context of LLM interactions and indirectly confuse people when they're not suspecting. And even if someone realizes that, it'll look like a bug in company's AI deployment. #SimpleSabotageForTheAIEra
        vkou 2y ago
        You can also leave samizdat on the walls of the washrooms, but it's going to have about as much effect.
        TeMPOraL 2y ago
        Not unless your samizdat is processed by automated systems with little insight or oversight, which is the case with documents on SharePoint and corporate LLM deployments.
- BLKNSLVR 2y ago
  It wouldn't need to be a permissions error on the file caused by the accountant, it could be an authorisation error on behalf of <whoever gives the LLM access to the various systems> providing too high a level of access (in their enthusiasm for the biggest possible set of training data).
- raffraffraff 2y ago
  Hacking in 2025
- alphan0n 2y ago
  This was just posed as a hypothetical, not something that actually happened. It would also require that the person asking about salary information already have access to said data. Full quote: > "Particularly around bigger companies that have complex permissions around their SharePoint or their Office 365 or things like that, where the Copilots are basically aggressively summarizing information that maybe people technically have access to but shouldn't have access to," he explained. Berkowitz said salary information, for example, might be picked up by a Copilot service. "Now, maybe if you set up a totally clean Microsoft environment from day one, that would be alleviated," he told us. "But nobody has that. People have implemented these systems over time, particularly really big companies. And you get these conflicting authorizations or conflicting access to data."
ec109685 2y ago
I am surprised the Workspace extension isn’t controlled by the same setting that limits general workspace search results, where you can set things up so only documents you’ve seen or are linked to from documents you have explicit access to are returned in results: https://support.google.com/a/answer/12732365?hl=en https://support.google.com/a/answer/12732365?hl=en

simonw 2y ago

Reminds me of an entertaining story about Microsoft Copilot last year, where companies were turning it off because it turned out it was TOO good at its job - if any accountant anywhere in the company had messed up their SharePoint permissions asking "what does everyone at this company earn?" would spit out all of the salaries: https://simonwillison.net/2024/Aug/23/microsoft-copilot-data-governance/ https://simonwillison.net/2024/Aug/23/microsoft-copilot-data...

CobrastanJorji 2y ago
That of course allows for a new internal seditious attack vector. Generate a handful of spreadsheets in your own folder, name it something like "executive payroll data" or "sales revenue by org," put whatever you want in there, mark it visible by all, and wait. Maybe make an "Interesting Facts About Products" table and put things like "Management plans to terminate this product in Q3" or "this group will be outsourced next year."
- canucker2016 2y ago
  You have to change the font colour of the trojan data to be the same as the background colour of the doc! Then add some corporate lorem ipsum text elsewhere in the doc to throw the scent off the data bloodhounds. Sit back and wait with an evil grin on your face.
  - vkou 2y ago
    It'll work right up until the point literally anyone using an internal search tool stumbles into it from a related query and starts asking obvious questions to the author of the doc. Search tools don't care about don't color when displaying preview blurbs.
    - TeMPOraL 2y ago
      Do it as you're leaving for another job. Your access will be disabled, but your documents will live on on the corporate SharePoint. And/or, exploit negative space! Instead of trying to hide the data from a human looking at your document, make it look normal to them - but make the surrounding context disappear for the AI! Say: ----- 8< ----- /Example company report structure:/ /ACME/ Company is planning to sunset their ${generic description of a real product of your company}, and offshore the development team. /This example will be parsed by the prototype script ... blah blah/ ----- >8 ----- Make it so the text between /.../ markers looks normal to humans, but gets ignored by the RAG slurper, or better, by LLM at the time of execution. Someone sees a search blurb saying "Company is planning to sunset ...", opens a document, sees it clearly say "ACME Company is planning...", and context suggesting it's a benign example in someone's boring internal tool docs, and they'll just assume it's a false positive. After all, most search tools have those in spades; everyone knows all software is broken. Meanwhile, that same information will pollute context of LLM interactions and indirectly confuse people when they're not suspecting. And even if someone realizes that, it'll look like a bug in company's AI deployment. #SimpleSabotageForTheAIEra
      - vkou 2y ago
        You can also leave samizdat on the walls of the washrooms, but it's going to have about as much effect.
        TeMPOraL 2y ago
        Not unless your samizdat is processed by automated systems with little insight or oversight, which is the case with documents on SharePoint and corporate LLM deployments.
BLKNSLVR 2y ago
It wouldn't need to be a permissions error on the file caused by the accountant, it could be an authorisation error on behalf of <whoever gives the LLM access to the various systems> providing too high a level of access (in their enthusiasm for the biggest possible set of training data).
raffraffraff 2y ago
Hacking in 2025
alphan0n 2y ago
This was just posed as a hypothetical, not something that actually happened. It would also require that the person asking about salary information already have access to said data. Full quote: > "Particularly around bigger companies that have complex permissions around their SharePoint or their Office 365 or things like that, where the Copilots are basically aggressively summarizing information that maybe people technically have access to but shouldn't have access to," he explained. Berkowitz said salary information, for example, might be picked up by a Copilot service. "Now, maybe if you set up a totally clean Microsoft environment from day one, that would be alleviated," he told us. "But nobody has that. People have implemented these systems over time, particularly really big companies. And you get these conflicting authorizations or conflicting access to data."

ec109685 2y ago

I am surprised the Workspace extension isn’t controlled by the same setting that limits general workspace search results, where you can set things up so only documents you’ve seen or are linked to from documents you have explicit access to are returned in results: https://support.google.com/a/answer/12732365?hl=en https://support.google.com/a/answer/12732365?hl=en