6 ms·

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to execut

by planb 1y ago

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. And as access rights are complex and can change at any given moment, that would still be prone to race conditions.

cryptonym 1y ago
True, per user doesn't scale. Knowledge should be properly grouped and have rights on database, documents, and chatbot managed by groups. For instance specific user can use the Engineering chatbot but not the Finance one. If you fail to define these groups, feels like you don't have a solid strategy. In the end, if that's what they want, let them experience open knowledge.
- planb 1y ago
  Yeah. If you have knowledge stored in a structured form like that, you don't need an AI...
  - cryptonym 1y ago
    If organisation is that bad that finance docs are mixed with engineering docs, how do you even onboard people? You manually go through every single doc and decide if the newcomer can or can't access it? You should see our Engineering knowledge base before saying an AI would be useless.
- 9dev 1y ago
  As if knowledge was ever that clear cut. Sometimes you need a cross-department insight, some data points from finance may not be confidential, some engineering content may be relevant to sales support… there’s endless reasons why neat little compartments like this don’t work in reality.
carschno 1y ago
> I am having a really hard time communicating this problem to executives When you hit such a wall, you might not be failing to communicate, nor them failing to understand. In reality, said executives have probably chosen to ignore the issue, but also don't want to take accountability for the eventual leaks. So "not understanding" is the easiest way to blame the engineers later.
- lupusreal 1y ago
  It doesn't even need to be blaming the engineers in this case, they can blame "the AI" and most people will accept that and let whatever incident happened slide. If somebody questions the wisdom of putting AI in such a position, they can be dismissed as not appreciating new technology (even though their concern is valid.)
  - deleted 1y ago
    [deleted]
  - jeltz 1y ago
    Yeah, it is usually not about blaming the engineers in my experience. It is about so they can make a descion they want to make without having to think too hard or take any accountability. If nobody knew at the time it was bad everyone can just act surprised and call it an accident and just go on with their lives making similar uninformed descisions. In their dream world the engineers would not know about it either. Edit: Maybe we should call this style vibe management. :D
  - Lu2025 1y ago
    Yep. AI is wonderful for IP laundering and accountability laundering (is this even a term? It is now!)
  - flir 1y ago
    "the AI did it" is going to be the new "somebody hacked my facebook account" I wish I had a way of ensuring culpability remains with the human who published the text, regardless of who/what authored it.
gloosx 1y ago
What do you mean a problem? It's an AI man. Just ask it what to do man. It's thinking, it's really really big thinking. Big thing which does all big thinking. The multi-modal reasoning deep big thinking bro. Security, permissions, thats so important for you?? We have AI. It does thinking. What else do you need?? Because two brains are better than one. Your backlog doesn’t stand a chance. Get speed when you need it. Depth when you don’t. Make one change. Copilot handles the rest. It's your code’s guardian angel. It's AI bro. AGI is coming tomorrow. Delegate like a boss. Access rights and all the complex things can wait.
krisoft 1y ago
> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the potentially relevant list of documents you check which ones can the current user access. You only pass the ones over to the LLM which the user can see. This is very similar to how banks provide phone based services. The operator on the other side of the line can only see your account details once you have authenticated yourself. They can't accidentally tell you someone else's account balance, because they themselves don't have access to it unless they typed in all the information you provide them to authenticate yourself. You can't trick the operator to provide you with someone else's account balance because they can't see the account balance of anyone without authenticating first.
- chaosite 1y ago
  > You search for matching documents in your vector database / index. Once you have found the potentially relevant list of documents you check which ones can the current user access. You only pass the ones over to the LLM which the user can see. Sometimes the potentially relevant list of documents itself is a leak all by itself.
  - redwood 1y ago
    But you process that list in a trusted audited app tier not in the client environment
    - lixtra 1y ago
      A naive approach could still leak information through side channels. E.g. if you search regularly for foobar, the answer might suddenly get slower if foobar appears more in the document base. Depending on the context it could be relevant.
      - 1y ago
inejge 1y ago
> I am having a really hard time communicating this problem to executives Cc Legal/Compliance could do wonders to their capacity to understand the problem. Caveat, of course, that the execs might be pissed off that some peon is placing roadblocks in the way of their buzzword-happy plan.
- planb 1y ago
  That would surely be a possible way, but I don't want to block anything, I just want reasonable expectations and a basic understanding of the problem on all sides.
  - jacquesm 1y ago
    If you start CC'ing legal or compliance on such issues you may very well need a planb.
    - Lu2025 1y ago
      This is correct. People on top are very much ego driven and don't forgive those who say no to them or make them look bad.
smarx007 1y ago
Two points/questions: 1. Why is tracking access rights "on a per user basis or [...] along with the content" is not feasible? A few mentions: Google Zanzibar (+Ory Keto as OSS impl) - makes authz for content othoronal to apps (i.e. possible to have it in one place, s.t. both Jira and a Jira MCP server can use the same API to check authz - possible to have a 100% faithful authz logic in the MCP server), Eclipse Biscuit (as far as I understand, this is a Dassault's attempt to make JWTs on steroids by adding Datalog and attenuation to the tokens, going in the Zanzibar direction but not requiring a network call for every single check), Apache Accumulo (DBMS with a cell-level security) and others. The way I see it, the tech is there but so far, not enough attention has been put on the problem of a high-fidelity authz throughout the enterprise on a granular level. 2. What is the scale needed? Enterprises with more than 10000 employees are quite rare, many individual internal IT systems even in large companies have less than 100 regular users. At these levels of scale, a lot more approaches are feasible that would not be considered possible at Google scale (i.e. more expensive algorithms w.r.t. big-O are viable).
- ForHackernews 1y ago
  As I understand it, there's no real way to enforce access rights inside an LLM. If the bot has access to some data, and you have access to the bot, you can potentially trick it into coughing up the data regardless of whether you're supposed to see that info or not.
  - smarx007 1y ago
    MCP tools with OAuth support + RFC 8693 OAuth 2.0 Token Exchange (aka OAuth 2.0 On-Behalf-Of flow in Azure Entra - though I don't think MCP 2025-06-18 accounts for the RFC 8693) could be used to limit the MCP bot responses to what the current user is authorized to see.
- PunchyHamster 1y ago
  Because the problem is not "get a list of what user can access" but "the AI that got trained on dataset must not leak to user that doesn't have access to it. There is no feasible way to track that during training (at least yet), so only current solution would be to learn AI agent only on data use can access and that is costly
everdrive 1y ago
>communicating this problem to executives I don't just mean this as lazy cynicism; executives don't really want to understand things. It doesn't suit their goals. They're not really in the business of strictly understanding things. They're in the business of "achieving success." And, in their world, a lot of success is really just the perception of success. Success and the perception of success are pretty interchangeable in their eyes, and they often feel that a lot of engineering concerns should really be dismissed unless those concerns are truly catastrophic.
- thewebguyd 1y ago
  > they often feel that a lot of engineering concerns should really be dismissed unless those concerns are truly catastrophic. Grizzled sysadmin here, and this is accurate. Classic case of "Hey boss, I need budget for server replacements, this hardware is going to fail." Declined. few months later, fails. Boss: "Why did you allow this to happen, what am I even paying you for?"
DannyBee 1y ago
"would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale" Citation needed. Most enterprise (homegrown or not) search engine products have to do this, and have been able to do it effectively at scale, for decades at this point. This is a very well known and well-solved problem, and the solutions are very directly applicable to the products you list. It is, as they say, a simple matter of implementation - if they don't offer it, it's because they haven't had the engineering time and/or customer need to do it. Not because it doesn't scale.
- malfist 1y ago
  If you're stringing together a bunch of MCPs you probably also have to string together a bunch of authorization mechanisms. Try having your search engine confirm live each persons access to each possible row. It's absolutely a hard problem and it isn't well solved
  - DannyBee 1y ago
    Yes, if you try to string together 30 systems with no controls and implement controls at the end it can be hard and slow - "this method i designed to not work doesn't work" is not very surprising. But the reply i made was to " This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale." IE information retrieval. Access control in information retrieval is a very well studied. Making search engines, etc that effectively confirm user access to each possible record is feasible and common (They don't do it exactly this way but the result is the same), and scalable. Hell, we even known how to do private information retrieval with access control in scalable ways. PIR = the server does not know what the query was, or the result was, but still retrieves the result. So we know how to make it so not only does the server does not know what was queried or retrieved by a user, but each querying user still only can access records they are allowed to. Overhead of this, which is much harder than non-private information retrieval with access control, is only 2-3x in computation. See, e.g., https://dspace.mit.edu/handle/1721.1/151392 https://dspace.mit.edu/handle/1721.1/151392 for one example of such a system. There are others. So even if your 2ms retrieval latency was all CPU and 0 I/O, it would only become 4-6ms do to this. If you remove the PIR part, as i said, it's much easier, and the overhead is much much less, since it doesn't involve tons and tons of computationally expensive encryption primitives (though some schemes still involve some).
giamma 1y ago
I believe most vector databases allow you to annotate vectors with additional metadata. Why not simply add as metadata the list of principals (roles/groups) who have access to the information (e.g. HR, executives) ? Then when a user makes a request to the chatbot, you expand the user identity to his/her principals (e.g. HR) and use those as implicit filtering criteria for finding the closest vectors in the database. In this way you exclude up-front the documents that the current user cannot see. Of course, this requires you to update the vector metadata any time the permissions change at the document level (e.g. a given document originally visible only to HR is now also visibile to executives -> you need to add the principal executives to the metadata of the vector resulting from the document in your vector database)
- sporkland 1y ago
  This is the correct answer. You do a pre-filter on a permissions correlated field like this and post-filter on the results for the deeper perms checks.
- planb 1y ago
  I am in control of the vector database and the search index. I have no control over the different accessed data sources that don’t even allow to query access rights per resource (and just allow for can_access checks for a given user)
awirth 1y ago
What you're describing is a specific case of a confused deputy problem: https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Confused_deputy_problem This is captured in the OWASP LLM Top 10 "LLM02:2025 Sensitive Information Disclosure" risk: https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/ https://genai.owasp.org/llmrisk/llm022025-sensitive-informat... although in some cases the "LLM06:2025 Excessive Agency" risk is also applicable. I believe that some enterprise RAG solutions create a per user index to solve this problem when there are lots of complex ACLs involved. How vendors manage this problem is an important question to ask when analyzing RAG solutions. At my current company at least we call this "権限混同" in Japanese - Literally "authorization confusion" which I think is a more fun name
- lmeyerov 1y ago
  Exactly. We often end up doing 'direct' retrieval (ex: DB query gen) to skip the time suck , costs , and insecurity of vector RAG, and per user indexing for the same. Agentic reasoning loops means this can be better quality and faster anyways. Sometimes hard to avoid though, like our firehose analyzers :(
sporkland 1y ago
If you have a field like and acl_id or some other context information on the data that is linked closely to a user's files. You can pass in the user's set of those field values to the vector database to pre-filter the results and do a permissions post check with a fairly relevant set. The vector db definitely has to do some heavy lifting intersecting the say acl_id normal index with the nearest neighbors search but they do support it.
AmazingTurtle 1y ago
No need for a per-user database, simply attach ACL to your vector DB (in my case I am use postgres, RLS for example or a baked ACL policy list if you're using opensearch for example)
8b7875ff 1y ago
> As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This is the way. This is also a solved problem. We solved it for desktop, web, mobile. Chatbots are just another untrusted frontend and should follow the same patterning to mitigate risks. I.E. do not trust inputs, use the same auth patterns you would for anything else (oauth, ect.). It is solved and not new.
bitfilped 1y ago
I know ethics aren't high up on the list of things we're taught about in tech, so I'd like to take a moment and point out that it's your moral responsibility to remove yourself from a project like this (or the company doing it.)
justincormack 1y ago
I listened to a podcast from someone at Glean about this, they do that, but she pointed out it is not enough as permissions are wrong often, and a good AI search can find you a bunch of documents with salary information that you have permission to see bit should not.
jsshapiro 1y ago
Depending on how you construct this, it may be a lot harder than you are saying. If your approach is to build a chatbot that scans the documents, you can enforce access on a per-session basis by limiting the documents available to the application. But if the approach is to train a neural net on this body of sensitive documents then you have a bigger problem. Actually two. The first is that the access control requirements have to be accounted for in the scoring function, which amounts to building a different engine for each user context. Though I suppose you could think of it as a composed neural net whose first net maps the input onto the [0,1] range on a per-user basis using the access control rules, and whose second net takes those results and runs them through additional layers. The second is that the trailing neural net won't converge the same way for different inputs, and (so far as I'm aware) there isn't any theory for how to propagate access restrictions across a neural net. Before inventing an old wheel, does anybody know of work on this in the research literature?
kephasp 1y ago
One typical way to resolve this is to use Voluntary Oblivious Compliance (VOC). In this capability-based pattern, every storage service could provide an opaque handler to a user that represents their authorization and it can be used to restrict on which documents indexing is done. http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Computing/Capability_Patterns#Oblivious_Claim_Check:_Loan_Officer_Protocol http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu... http://www.skyhunter.com/marcs/ewalnut.html#proofOfPurchase http://www.skyhunter.com/marcs/ewalnut.html#proofOfPurchase If the opaque handle is part of the Membrane pattern, you can even avoid most race conditions, because even during the indexing, the capabilities can be used to access documents and that removes the possibility of a TOCTOU race. http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Computing/Capability_Patterns#Membranes http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu...

by planb 1y ago

cryptonym 1y ago

True, per user doesn't scale. Knowledge should be properly grouped and have rights on database, documents, and chatbot managed by groups. For instance specific user can use the Engineering chatbot but not the Finance one. If you fail to define these groups, feels like you don't have a solid strategy. In the end, if that's what they want, let them experience open knowledge.

planb 1y ago
Yeah. If you have knowledge stored in a structured form like that, you don't need an AI...
- cryptonym 1y ago
  If organisation is that bad that finance docs are mixed with engineering docs, how do you even onboard people? You manually go through every single doc and decide if the newcomer can or can't access it? You should see our Engineering knowledge base before saying an AI would be useless.
9dev 1y ago
As if knowledge was ever that clear cut. Sometimes you need a cross-department insight, some data points from finance may not be confidential, some engineering content may be relevant to sales support… there’s endless reasons why neat little compartments like this don’t work in reality.

carschno 1y ago

> I am having a really hard time communicating this problem to executives When you hit such a wall, you might not be failing to communicate, nor them failing to understand. In reality, said executives have probably chosen to ignore the issue, but also don't want to take accountability for the eventual leaks. So "not understanding" is the easiest way to blame the engineers later.

lupusreal 1y ago
It doesn't even need to be blaming the engineers in this case, they can blame "the AI" and most people will accept that and let whatever incident happened slide. If somebody questions the wisdom of putting AI in such a position, they can be dismissed as not appreciating new technology (even though their concern is valid.)
- deleted 1y ago
  [deleted]
- jeltz 1y ago
  Yeah, it is usually not about blaming the engineers in my experience. It is about so they can make a descion they want to make without having to think too hard or take any accountability. If nobody knew at the time it was bad everyone can just act surprised and call it an accident and just go on with their lives making similar uninformed descisions. In their dream world the engineers would not know about it either. Edit: Maybe we should call this style vibe management. :D
- Lu2025 1y ago
  Yep. AI is wonderful for IP laundering and accountability laundering (is this even a term? It is now!)
- flir 1y ago
  "the AI did it" is going to be the new "somebody hacked my facebook account" I wish I had a way of ensuring culpability remains with the human who published the text, regardless of who/what authored it.

gloosx 1y ago

What do you mean a problem? It's an AI man. Just ask it what to do man. It's thinking, it's really really big thinking. Big thing which does all big thinking. The multi-modal reasoning deep big thinking bro. Security, permissions, thats so important for you?? We have AI. It does thinking. What else do you need?? Because two brains are better than one. Your backlog doesn’t stand a chance. Get speed when you need it. Depth when you don’t. Make one change. Copilot handles the rest. It's your code’s guardian angel. It's AI bro. AGI is coming tomorrow. Delegate like a boss. Access rights and all the complex things can wait.

krisoft 1y ago

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the potentially relevant list of documents you check which ones can the current user access. You only pass the ones over to the LLM which the user can see. This is very similar to how banks provide phone based services. The operator on the other side of the line can only see your account details once you have authenticated yourself. They can't accidentally tell you someone else's account balance, because they themselves don't have access to it unless they typed in all the information you provide them to authenticate yourself. You can't trick the operator to provide you with someone else's account balance because they can't see the account balance of anyone without authenticating first.

chaosite 1y ago
> You search for matching documents in your vector database / index. Once you have found the potentially relevant list of documents you check which ones can the current user access. You only pass the ones over to the LLM which the user can see. Sometimes the potentially relevant list of documents itself is a leak all by itself.
- redwood 1y ago
  But you process that list in a trusted audited app tier not in the client environment
  - lixtra 1y ago
    A naive approach could still leak information through side channels. E.g. if you search regularly for foobar, the answer might suddenly get slower if foobar appears more in the document base. Depending on the context it could be relevant.
    - 1y ago

inejge 1y ago

> I am having a really hard time communicating this problem to executives Cc Legal/Compliance could do wonders to their capacity to understand the problem. Caveat, of course, that the execs might be pissed off that some peon is placing roadblocks in the way of their buzzword-happy plan.

planb 1y ago
That would surely be a possible way, but I don't want to block anything, I just want reasonable expectations and a basic understanding of the problem on all sides.
- jacquesm 1y ago
  If you start CC'ing legal or compliance on such issues you may very well need a planb.
  - Lu2025 1y ago
    This is correct. People on top are very much ego driven and don't forgive those who say no to them or make them look bad.

smarx007 1y ago

Two points/questions: 1. Why is tracking access rights "on a per user basis or [...] along with the content" is not feasible? A few mentions: Google Zanzibar (+Ory Keto as OSS impl) - makes authz for content othoronal to apps (i.e. possible to have it in one place, s.t. both Jira and a Jira MCP server can use the same API to check authz - possible to have a 100% faithful authz logic in the MCP server), Eclipse Biscuit (as far as I understand, this is a Dassault's attempt to make JWTs on steroids by adding Datalog and attenuation to the tokens, going in the Zanzibar direction but not requiring a network call for every single check), Apache Accumulo (DBMS with a cell-level security) and others. The way I see it, the tech is there but so far, not enough attention has been put on the problem of a high-fidelity authz throughout the enterprise on a granular level. 2. What is the scale needed? Enterprises with more than 10000 employees are quite rare, many individual internal IT systems even in large companies have less than 100 regular users. At these levels of scale, a lot more approaches are feasible that would not be considered possible at Google scale (i.e. more expensive algorithms w.r.t. big-O are viable).

ForHackernews 1y ago
As I understand it, there's no real way to enforce access rights inside an LLM. If the bot has access to some data, and you have access to the bot, you can potentially trick it into coughing up the data regardless of whether you're supposed to see that info or not.
- smarx007 1y ago
  MCP tools with OAuth support + RFC 8693 OAuth 2.0 Token Exchange (aka OAuth 2.0 On-Behalf-Of flow in Azure Entra - though I don't think MCP 2025-06-18 accounts for the RFC 8693) could be used to limit the MCP bot responses to what the current user is authorized to see.
PunchyHamster 1y ago
Because the problem is not "get a list of what user can access" but "the AI that got trained on dataset must not leak to user that doesn't have access to it. There is no feasible way to track that during training (at least yet), so only current solution would be to learn AI agent only on data use can access and that is costly

everdrive 1y ago

>communicating this problem to executives I don't just mean this as lazy cynicism; executives don't really want to understand things. It doesn't suit their goals. They're not really in the business of strictly understanding things. They're in the business of "achieving success." And, in their world, a lot of success is really just the perception of success. Success and the perception of success are pretty interchangeable in their eyes, and they often feel that a lot of engineering concerns should really be dismissed unless those concerns are truly catastrophic.

thewebguyd 1y ago
> they often feel that a lot of engineering concerns should really be dismissed unless those concerns are truly catastrophic. Grizzled sysadmin here, and this is accurate. Classic case of "Hey boss, I need budget for server replacements, this hardware is going to fail." Declined. few months later, fails. Boss: "Why did you allow this to happen, what am I even paying you for?"

DannyBee 1y ago

"would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale" Citation needed. Most enterprise (homegrown or not) search engine products have to do this, and have been able to do it effectively at scale, for decades at this point. This is a very well known and well-solved problem, and the solutions are very directly applicable to the products you list. It is, as they say, a simple matter of implementation - if they don't offer it, it's because they haven't had the engineering time and/or customer need to do it. Not because it doesn't scale.

malfist 1y ago
If you're stringing together a bunch of MCPs you probably also have to string together a bunch of authorization mechanisms. Try having your search engine confirm live each persons access to each possible row. It's absolutely a hard problem and it isn't well solved
- DannyBee 1y ago
  Yes, if you try to string together 30 systems with no controls and implement controls at the end it can be hard and slow - "this method i designed to not work doesn't work" is not very surprising. But the reply i made was to " This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale." IE information retrieval. Access control in information retrieval is a very well studied. Making search engines, etc that effectively confirm user access to each possible record is feasible and common (They don't do it exactly this way but the result is the same), and scalable. Hell, we even known how to do private information retrieval with access control in scalable ways. PIR = the server does not know what the query was, or the result was, but still retrieves the result. So we know how to make it so not only does the server does not know what was queried or retrieved by a user, but each querying user still only can access records they are allowed to. Overhead of this, which is much harder than non-private information retrieval with access control, is only 2-3x in computation. See, e.g., https://dspace.mit.edu/handle/1721.1/151392 https://dspace.mit.edu/handle/1721.1/151392 for one example of such a system. There are others. So even if your 2ms retrieval latency was all CPU and 0 I/O, it would only become 4-6ms do to this. If you remove the PIR part, as i said, it's much easier, and the overhead is much much less, since it doesn't involve tons and tons of computationally expensive encryption primitives (though some schemes still involve some).

giamma 1y ago

I believe most vector databases allow you to annotate vectors with additional metadata. Why not simply add as metadata the list of principals (roles/groups) who have access to the information (e.g. HR, executives) ? Then when a user makes a request to the chatbot, you expand the user identity to his/her principals (e.g. HR) and use those as implicit filtering criteria for finding the closest vectors in the database. In this way you exclude up-front the documents that the current user cannot see. Of course, this requires you to update the vector metadata any time the permissions change at the document level (e.g. a given document originally visible only to HR is now also visibile to executives -> you need to add the principal executives to the metadata of the vector resulting from the document in your vector database)

sporkland 1y ago
This is the correct answer. You do a pre-filter on a permissions correlated field like this and post-filter on the results for the deeper perms checks.
planb 1y ago
I am in control of the vector database and the search index. I have no control over the different accessed data sources that don’t even allow to query access rights per resource (and just allow for can_access checks for a given user)

awirth 1y ago

What you're describing is a specific case of a confused deputy problem: https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Confused_deputy_problem This is captured in the OWASP LLM Top 10 "LLM02:2025 Sensitive Information Disclosure" risk: https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/ https://genai.owasp.org/llmrisk/llm022025-sensitive-informat... although in some cases the "LLM06:2025 Excessive Agency" risk is also applicable. I believe that some enterprise RAG solutions create a per user index to solve this problem when there are lots of complex ACLs involved. How vendors manage this problem is an important question to ask when analyzing RAG solutions. At my current company at least we call this "権限混同" in Japanese - Literally "authorization confusion" which I think is a more fun name

lmeyerov 1y ago
Exactly. We often end up doing 'direct' retrieval (ex: DB query gen) to skip the time suck , costs , and insecurity of vector RAG, and per user indexing for the same. Agentic reasoning loops means this can be better quality and faster anyways. Sometimes hard to avoid though, like our firehose analyzers :(

sporkland 1y ago

If you have a field like and acl_id or some other context information on the data that is linked closely to a user's files. You can pass in the user's set of those field values to the vector database to pre-filter the results and do a permissions post check with a fairly relevant set. The vector db definitely has to do some heavy lifting intersecting the say acl_id normal index with the nearest neighbors search but they do support it.

AmazingTurtle 1y ago

No need for a per-user database, simply attach ACL to your vector DB (in my case I am use postgres, RLS for example or a baked ACL policy list if you're using opensearch for example)

8b7875ff 1y ago

> As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This is the way. This is also a solved problem. We solved it for desktop, web, mobile. Chatbots are just another untrusted frontend and should follow the same patterning to mitigate risks. I.E. do not trust inputs, use the same auth patterns you would for anything else (oauth, ect.). It is solved and not new.

bitfilped 1y ago

I know ethics aren't high up on the list of things we're taught about in tech, so I'd like to take a moment and point out that it's your moral responsibility to remove yourself from a project like this (or the company doing it.)

justincormack 1y ago

I listened to a podcast from someone at Glean about this, they do that, but she pointed out it is not enough as permissions are wrong often, and a good AI search can find you a bunch of documents with salary information that you have permission to see bit should not.

jsshapiro 1y ago

Depending on how you construct this, it may be a lot harder than you are saying. If your approach is to build a chatbot that scans the documents, you can enforce access on a per-session basis by limiting the documents available to the application. But if the approach is to train a neural net on this body of sensitive documents then you have a bigger problem. Actually two. The first is that the access control requirements have to be accounted for in the scoring function, which amounts to building a different engine for each user context. Though I suppose you could think of it as a composed neural net whose first net maps the input onto the [0,1] range on a per-user basis using the access control rules, and whose second net takes those results and runs them through additional layers. The second is that the trailing neural net won't converge the same way for different inputs, and (so far as I'm aware) there isn't any theory for how to propagate access restrictions across a neural net. Before inventing an old wheel, does anybody know of work on this in the research literature?

kephasp 1y ago

One typical way to resolve this is to use Voluntary Oblivious Compliance (VOC). In this capability-based pattern, every storage service could provide an opaque handler to a user that represents their authorization and it can be used to restrict on which documents indexing is done. http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Computing/Capability_Patterns#Oblivious_Claim_Check:_Loan_Officer_Protocol http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu... http://www.skyhunter.com/marcs/ewalnut.html#proofOfPurchase http://www.skyhunter.com/marcs/ewalnut.html#proofOfPurchase If the opaque handle is part of the Membrane pattern, you can even avoid most race conditions, because even during the indexing, the capabilities can be used to access documents and that removes the possibility of a TOCTOU race. http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Computing/Capability_Patterns#Membranes http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu...