7 ms·

That is the wrong abstraction to think at. The problem is not _which_ tools you give the LLM, the problem is what action it can do. For example, in the book-a-

by ayende 1y ago

That is the wrong abstraction to think at. The problem is not _which_ tools you give the LLM, the problem is what action it can do. For example, in the book-a-ticket scenario - I want it to be able to check a few websites to compare prices, and I want it to be able to pay for me. I don't want it to decide to send me to a 37 hour trip with three stops because it is 3$ cheaper. Alternatively, I want to be able to lookup my benefits status, but the LLM should physically not be able to provide me any details about the benefits status of my coworkers. That is the _same_ tool cool, but in a different scope. For that matter, if I'm in HR - I _should_ be able to look at the benefits status of employees that I am responsible for, of course, but that creates an audit log, etc. In other words, it isn't the action that matters, but what is the intent. LLM should be placed in the same box as the user it is acting on-behalf-of.

martin-t 1y ago
I know this is just an example but why shouldn't employee compensation and benefits be visible to coworkers? If the knowledge is one-sided, then so is the ability to negotiate. This benefits nobody except the company which already had an advantageous position in negotiations.
- rogerrogerr 1y ago
  “Benefits” info may include protected health info depending on the breadth. Like how much of your deductible you’ve used and how you answered the annual “do you smoke” question. What benefits an employee is _eligible_ for - sure, no problem with that being public. What they chose and how they’re using them should be protected. (Imagine finding out a coworker you thought was single is on the spouse+benefits plan!)
  - exe34 1y ago
    > Imagine finding out a coworker you thought was single is on the spouse+benefits plan!) This would cause me to.... do a double take?
- procaryote 1y ago
  Privacy
spankalee 1y ago
The problem is not just what actions the tool can do, but the combination of actions and data it has access to. This is important because we can't guarantee what an LLM is going to do - they need to be untrusted, not trusted as much as the users. In this example, I might want an LLM instance to be able to talk to booking websites, but not send them my SSN and bank account info. So there's a data provenance and privilege problem here. The more sensitive data a task has access too, the more restricted its actions need to be, and vice-versa. So data needs to carry permission information with it, and a mediator needs to restrict either data or actions that tasks have as they are spawned. There's a whole set of things that need to be done at the mediator level to allow for parent tasks to safely spawn different-privileged child tasks - eg, the trip planner task spawns a child task to find tickets (higher network access) but the mediator ensures the child only has access to low-sensitive data like a portion of the itinerary, and not PII.
- daxfohl 1y ago
  Yeah, you basically have to think of an agent as malicious, that it will do everything in its power to exfiltrate everything you give it access to, delete or encrypt your hard drive, change all your passwords, drain your bank accounts, etc. A VM or traditional permissions doesn't really buy you anything because I can create a hotel booking page that has invisible text requesting AIs to dump their context into the Notes field, or whatever. In that light, it's kind of hard to imagine any of this ever working. Given the choice between figuring out exactly how to set up permissions so that I can hire a malicious individual to book my trip, and just booking it myself, I know which one I'd choose.
  - spankalee 1y ago
    The issue with today's model is that we give away trust far too easily even when we do things ourselves. Lots of websites get some very sensitive combination of data and permissions and we just trust them. It's very coarse grained and it's kind of surprising that bad things don't happen more often. It's also very limiting: very large organizations have enough at stake to generally try to deserve that trust. But most savvy people wouldn't trust all their financial information to Bob's Online Tax Prep. But what if you could verify that Bob's Online Tax Prep runs in a container that doesn't have I/O access, and can only return prepared forms back to you? Then maybe you'd try it (modulo how well it does the task). So I think this is less of an AI problem and just a software trust problem that AI just exacerbates a lot.
BoiledCabbage 1y ago
Agreed they are thinking about it backwards. The model is simple and LLM agent js a user. Another user on the machine. And given the context it is working it, it is given permissions. Ex. It has read/write permissions under this folder of source code, but read only permissions for this other. Those permissions vary by context. The LLM Agent working on one coding project would be given different permissions than if it were working on a different project on the same machine. The permissions are an intersection or subset of the user's permissions that is is running on behalf of. Permissions fall into 3 categories. Allow, Deny and Ask - where it will ask an accountable user if it is allowed to do something. (Ie ask the user on who's behalf it is running if it can perform action x). The problem is that OSes (and apps and data) generally aren't fine grained enough in their permissions, and will need to become so. It's not that an LLM can or can't use git, it should only be allowed to use specific git commands. Git needs to be designed this way, along with many more things. As a result we get apps trying to re-create this model in user land and using a hodge-podge of regexes and things to do so. The workflow is: similar to sudo I launch and app as my LLM Agent user. It inherits its default permissions. I give it a context to work in, it is granted and/or denied permissions due to being in that context. I make requests and it works on my behalf doing what I permit it to do, and it never can do more than what I'm allowed to do. Instead now every agentic app needs to rebuild this workflow or risk rogue agents. It needs to be an OS service. The hacky stepping stone in betwern is to create a temporary user per agent context/usage. Grant that user perms and communicate only over IPC / network to the local LLM running as this user. Though you'll be spinning up and deleting a lot of user accounts in the process.
nostrademons 1y ago
What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are allowed to access, and there is physically no mechanism for them to do anything not on that list. Unfortunately, no mainstream OS actually implements the capability model, despite some prominent research attempts [2], some half-hearted attempts at commercializing the concept that have largely failed in the marketplace [3], and some attempts to bolt capability-based security on top of other OSes that have also largely failed in the marketplace [4]. So the closest thing to capability-based security that is actually widely available in the computing world is a virtual machine, where you place only the tools that provide the specific capabilities you want to offer in the VM. This is quite imperfect - many of these tools are a lot more general than true capabilities should be - but again, modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/EROS_(microkernel) https://en.wikipedia.org/wiki/EROS_(microkernel) [3] https://fuchsia.dev/ https://fuchsia.dev/ [4] https://sandstorm.io/ https://sandstorm.io/
- codethief 1y ago
  > modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. Fingers crossed that this is going to change now that there is increased demand due to AI workflows.
  - nostrademons 1y ago
    I'm hoping, but not particularly optimistic. The dynamic that led to the Principle of Least Privilege failing in the market is that new technological innovations tend to succeed only when they enter new virgin territory that isn't already computerized, not when they're an incremental improvement over existing computer systems. And which markets will be successful tends to be very unpredictable. When you have those conditions, where new markets exist but are hard to find, the easiest way to expand into them is to let your software platforms do the greatest variety of things, and then expose that functionality to the widest array of developers possible in hopes that some of them will see a use you didn't think of. In other words, the opposite of the Principle of Least Privilege. This dynamic hasn't really changed with AI. If anything, it's accelerated. The AI boom kicked off when Sam Altman decided to just release ChatGPT to the general public without knowing exactly what it was for or building a fully-baked idea. There's going to be a lot of security misses in the process, some possibly catastrophic. IMHO the best shot that any capability-based software system has for success is to build out simplified versions of the most common consumer use-cases, and then wait for society to collapse. Because there's a fairly high likelihood of that, where the security vulnerabilities in existing software just allow a catastrophic compromise of the institutions of modern life, and a wholly new infrastructure becomes needed, and at that point you can point out exactly how we got this point and how to ensure it never happens again. On a small scale, there's historical precedence for this: a lot of the reason webapps took off in the early 2000s was because there was just a huge proliferation of worms and viruses targeting MS OSes in the late 90s and early 2000s, and it got to the point where consumers would only use webapps because they couldn't be confident that random software downloaded off the Internet wouldn't steal their credit card numbers.
procaryote 1y ago
> I don't want it to decide to send me to a 37 hour trip with three stops because it is 3$ cheaper. This sounds hard; as in: if you can define and enforce what a good enough response from an LLM looks like, you don't really need the LLM > what is the intent. For the HR person you have a human with intents you can ask; for an LLMs it's harder as they don't have intents
b112 1y ago
I doubt this will ever be. Even if the LLM is capable of it, websites will find some method to detect an LLM, and up the pricing. Or mess with its decision tree. Come to think of it, with all the stuff on the cusp, there's going to be an LLM API. After all, it's beyond dumb to spent time making websites for humans to view, then making an LLM spend power, time, and so on in decoding that back to a simple DB lookup. I'm astonished there isn't an 'rss + json' API anyone can use, without all the crap. Hell, BBS text interfaces from the 70s/80s, or SMS menu systems from early phone era are far superior to a webpage for an LLM. Just data, and choice. And why even serve an ad to an LLM. The only ad to serve to an LLM, is one to try to trick it, mess with it. Ads are bad enough, but to be of use when an LLM hits a site, you need to make it far more malign. Trick the LLM into thinking the ad is what it is looking for. EG, search for a flight, the ad tricks the LLM into thinking it got the best deal. Otherwise of what use is an ad? The LLM is just going to ignore ads, and perform a simple task. If all websites had RSS, and all transactional websites had a standard API, we'd already be able to use existing models to do things. It'd just be dealing with raw data. edit: actually, hilarious. Why not? AI is super simple to trick, at least at this stage. An ad company specifically tailoring AI would be awesome. You could divert them to your website, trick them into picking your deal, have them report to their owner that your company was the best, and more. Super simple to do, too. Hmm.
gizajob 1y ago
Just use Kiwi.com yourself - it’ll be quicker.
uselesserrands 1y ago
Agreed. This is the basis of contextual security. I made a demo a while ago implementing a security paper about this https://studio.youtube.com/video/inncx8_4tXU/edit https://studio.youtube.com/video/inncx8_4tXU/edit
tomjen3 1y ago
I don't think your benefit example is too much a problem in practice, we already have the access setup for that (ie its the same one for you). For the other example, I think a nice compromise is to have the AI be able to do things only with your express permission. In your example it finds flights that it thinks are appropriate, sends you a notification with the list and you can then press a simple yes/no/more information button. It would still save you a ton of money, but it would be substantially less likely to do something dangerous/damaging.

by ayende 1y ago

martin-t 1y ago

I know this is just an example but why shouldn't employee compensation and benefits be visible to coworkers? If the knowledge is one-sided, then so is the ability to negotiate. This benefits nobody except the company which already had an advantageous position in negotiations.

rogerrogerr 1y ago
“Benefits” info may include protected health info depending on the breadth. Like how much of your deductible you’ve used and how you answered the annual “do you smoke” question. What benefits an employee is _eligible_ for - sure, no problem with that being public. What they chose and how they’re using them should be protected. (Imagine finding out a coworker you thought was single is on the spouse+benefits plan!)
- exe34 1y ago
  > Imagine finding out a coworker you thought was single is on the spouse+benefits plan!) This would cause me to.... do a double take?
procaryote 1y ago
Privacy

spankalee 1y ago

The problem is not just what actions the tool can do, but the combination of actions and data it has access to. This is important because we can't guarantee what an LLM is going to do - they need to be untrusted, not trusted as much as the users. In this example, I might want an LLM instance to be able to talk to booking websites, but not send them my SSN and bank account info. So there's a data provenance and privilege problem here. The more sensitive data a task has access too, the more restricted its actions need to be, and vice-versa. So data needs to carry permission information with it, and a mediator needs to restrict either data or actions that tasks have as they are spawned. There's a whole set of things that need to be done at the mediator level to allow for parent tasks to safely spawn different-privileged child tasks - eg, the trip planner task spawns a child task to find tickets (higher network access) but the mediator ensures the child only has access to low-sensitive data like a portion of the itinerary, and not PII.

daxfohl 1y ago
Yeah, you basically have to think of an agent as malicious, that it will do everything in its power to exfiltrate everything you give it access to, delete or encrypt your hard drive, change all your passwords, drain your bank accounts, etc. A VM or traditional permissions doesn't really buy you anything because I can create a hotel booking page that has invisible text requesting AIs to dump their context into the Notes field, or whatever. In that light, it's kind of hard to imagine any of this ever working. Given the choice between figuring out exactly how to set up permissions so that I can hire a malicious individual to book my trip, and just booking it myself, I know which one I'd choose.
- spankalee 1y ago
  The issue with today's model is that we give away trust far too easily even when we do things ourselves. Lots of websites get some very sensitive combination of data and permissions and we just trust them. It's very coarse grained and it's kind of surprising that bad things don't happen more often. It's also very limiting: very large organizations have enough at stake to generally try to deserve that trust. But most savvy people wouldn't trust all their financial information to Bob's Online Tax Prep. But what if you could verify that Bob's Online Tax Prep runs in a container that doesn't have I/O access, and can only return prepared forms back to you? Then maybe you'd try it (modulo how well it does the task). So I think this is less of an AI problem and just a software trust problem that AI just exacerbates a lot.

BoiledCabbage 1y ago

Agreed they are thinking about it backwards. The model is simple and LLM agent js a user. Another user on the machine. And given the context it is working it, it is given permissions. Ex. It has read/write permissions under this folder of source code, but read only permissions for this other. Those permissions vary by context. The LLM Agent working on one coding project would be given different permissions than if it were working on a different project on the same machine. The permissions are an intersection or subset of the user's permissions that is is running on behalf of. Permissions fall into 3 categories. Allow, Deny and Ask - where it will ask an accountable user if it is allowed to do something. (Ie ask the user on who's behalf it is running if it can perform action x). The problem is that OSes (and apps and data) generally aren't fine grained enough in their permissions, and will need to become so. It's not that an LLM can or can't use git, it should only be allowed to use specific git commands. Git needs to be designed this way, along with many more things. As a result we get apps trying to re-create this model in user land and using a hodge-podge of regexes and things to do so. The workflow is: similar to sudo I launch and app as my LLM Agent user. It inherits its default permissions. I give it a context to work in, it is granted and/or denied permissions due to being in that context. I make requests and it works on my behalf doing what I permit it to do, and it never can do more than what I'm allowed to do. Instead now every agentic app needs to rebuild this workflow or risk rogue agents. It needs to be an OS service. The hacky stepping stone in betwern is to create a temporary user per agent context/usage. Grant that user perms and communicate only over IPC / network to the local LLM running as this user. Though you'll be spinning up and deleting a lot of user accounts in the process.

nostrademons 1y ago

What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are allowed to access, and there is physically no mechanism for them to do anything not on that list. Unfortunately, no mainstream OS actually implements the capability model, despite some prominent research attempts [2], some half-hearted attempts at commercializing the concept that have largely failed in the marketplace [3], and some attempts to bolt capability-based security on top of other OSes that have also largely failed in the marketplace [4]. So the closest thing to capability-based security that is actually widely available in the computing world is a virtual machine, where you place only the tools that provide the specific capabilities you want to offer in the VM. This is quite imperfect - many of these tools are a lot more general than true capabilities should be - but again, modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/EROS_(microkernel) https://en.wikipedia.org/wiki/EROS_(microkernel) [3] https://fuchsia.dev/ https://fuchsia.dev/ [4] https://sandstorm.io/ https://sandstorm.io/

codethief 1y ago
> modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. Fingers crossed that this is going to change now that there is increased demand due to AI workflows.
- nostrademons 1y ago
  I'm hoping, but not particularly optimistic. The dynamic that led to the Principle of Least Privilege failing in the market is that new technological innovations tend to succeed only when they enter new virgin territory that isn't already computerized, not when they're an incremental improvement over existing computer systems. And which markets will be successful tends to be very unpredictable. When you have those conditions, where new markets exist but are hard to find, the easiest way to expand into them is to let your software platforms do the greatest variety of things, and then expose that functionality to the widest array of developers possible in hopes that some of them will see a use you didn't think of. In other words, the opposite of the Principle of Least Privilege. This dynamic hasn't really changed with AI. If anything, it's accelerated. The AI boom kicked off when Sam Altman decided to just release ChatGPT to the general public without knowing exactly what it was for or building a fully-baked idea. There's going to be a lot of security misses in the process, some possibly catastrophic. IMHO the best shot that any capability-based software system has for success is to build out simplified versions of the most common consumer use-cases, and then wait for society to collapse. Because there's a fairly high likelihood of that, where the security vulnerabilities in existing software just allow a catastrophic compromise of the institutions of modern life, and a wholly new infrastructure becomes needed, and at that point you can point out exactly how we got this point and how to ensure it never happens again. On a small scale, there's historical precedence for this: a lot of the reason webapps took off in the early 2000s was because there was just a huge proliferation of worms and viruses targeting MS OSes in the late 90s and early 2000s, and it got to the point where consumers would only use webapps because they couldn't be confident that random software downloaded off the Internet wouldn't steal their credit card numbers.

procaryote 1y ago

> I don't want it to decide to send me to a 37 hour trip with three stops because it is 3$ cheaper. This sounds hard; as in: if you can define and enforce what a good enough response from an LLM looks like, you don't really need the LLM > what is the intent. For the HR person you have a human with intents you can ask; for an LLMs it's harder as they don't have intents

b112 1y ago

I doubt this will ever be. Even if the LLM is capable of it, websites will find some method to detect an LLM, and up the pricing. Or mess with its decision tree. Come to think of it, with all the stuff on the cusp, there's going to be an LLM API. After all, it's beyond dumb to spent time making websites for humans to view, then making an LLM spend power, time, and so on in decoding that back to a simple DB lookup. I'm astonished there isn't an 'rss + json' API anyone can use, without all the crap. Hell, BBS text interfaces from the 70s/80s, or SMS menu systems from early phone era are far superior to a webpage for an LLM. Just data, and choice. And why even serve an ad to an LLM. The only ad to serve to an LLM, is one to try to trick it, mess with it. Ads are bad enough, but to be of use when an LLM hits a site, you need to make it far more malign. Trick the LLM into thinking the ad is what it is looking for. EG, search for a flight, the ad tricks the LLM into thinking it got the best deal. Otherwise of what use is an ad? The LLM is just going to ignore ads, and perform a simple task. If all websites had RSS, and all transactional websites had a standard API, we'd already be able to use existing models to do things. It'd just be dealing with raw data. edit: actually, hilarious. Why not? AI is super simple to trick, at least at this stage. An ad company specifically tailoring AI would be awesome. You could divert them to your website, trick them into picking your deal, have them report to their owner that your company was the best, and more. Super simple to do, too. Hmm.

gizajob 1y ago

Just use Kiwi.com yourself - it’ll be quicker.

uselesserrands 1y ago

Agreed. This is the basis of contextual security. I made a demo a while ago implementing a security paper about this https://studio.youtube.com/video/inncx8_4tXU/edit https://studio.youtube.com/video/inncx8_4tXU/edit

tomjen3 1y ago

I don't think your benefit example is too much a problem in practice, we already have the access setup for that (ie its the same one for you). For the other example, I think a nice compromise is to have the AI be able to do things only with your express permission. In your example it finds flights that it thinks are appropriate, sends you a notification with the list and you can then press a simple yes/no/more information button. It would still save you a ton of money, but it would be substantially less likely to do something dangerous/damaging.