3 ms·

What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are a

What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are allowed to access, and there is physically no mechanism for them to do anything not on that list. Unfortunately, no mainstream OS actually implements the capability model, despite some prominent research attempts [2], some half-hearted attempts at commercializing the concept that have largely failed in the marketplace [3], and some attempts to bolt capability-based security on top of other OSes that have also largely failed in the marketplace [4]. So the closest thing to capability-based security that is actually widely available in the computing world is a virtual machine, where you place only the tools that provide the specific capabilities you want to offer in the VM. This is quite imperfect - many of these tools are a lot more general than true capabilities should be - but again, modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/EROS_(microkernel) https://en.wikipedia.org/wiki/EROS_(microkernel) [3] https://fuchsia.dev/ https://fuchsia.dev/ [4] https://sandstorm.io/ https://sandstorm.io/

codethief 1y ago
> modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. Fingers crossed that this is going to change now that there is increased demand due to AI workflows.
- nostrademons 1y ago
  I'm hoping, but not particularly optimistic. The dynamic that led to the Principle of Least Privilege failing in the market is that new technological innovations tend to succeed only when they enter new virgin territory that isn't already computerized, not when they're an incremental improvement over existing computer systems. And which markets will be successful tends to be very unpredictable. When you have those conditions, where new markets exist but are hard to find, the easiest way to expand into them is to let your software platforms do the greatest variety of things, and then expose that functionality to the widest array of developers possible in hopes that some of them will see a use you didn't think of. In other words, the opposite of the Principle of Least Privilege. This dynamic hasn't really changed with AI. If anything, it's accelerated. The AI boom kicked off when Sam Altman decided to just release ChatGPT to the general public without knowing exactly what it was for or building a fully-baked idea. There's going to be a lot of security misses in the process, some possibly catastrophic. IMHO the best shot that any capability-based software system has for success is to build out simplified versions of the most common consumer use-cases, and then wait for society to collapse. Because there's a fairly high likelihood of that, where the security vulnerabilities in existing software just allow a catastrophic compromise of the institutions of modern life, and a wholly new infrastructure becomes needed, and at that point you can point out exactly how we got this point and how to ensure it never happens again. On a small scale, there's historical precedence for this: a lot of the reason webapps took off in the early 2000s was because there was just a huge proliferation of worms and viruses targeting MS OSes in the late 90s and early 2000s, and it got to the point where consumers would only use webapps because they couldn't be confident that random software downloaded off the Internet wouldn't steal their credit card numbers.
  - black_knight 1y ago
    Sometimes better ideas can be around for a really long time before they gain any mainstream traction. Some ideas which come to mind are anonymous functions and sum types with pattern matching, which are only recently finding their way into mainstream languages, despite having been around for ages. What it might take is a dedicated effort over time by a group of believers, to keep the ideas alive and create new attempts, new projects regularly. So that when there is a mainstream opening, there is the knownhow to implement them. I always include a lecture or two in my software security course (150 students per year), on capability based security. I am also on the lookout for projects which could use the ideas, but so far I have only vague ideas that they could be combined with algebraic effects in some way in functional programming.
    - codethief 1y ago
      > but so far I have only vague ideas that they could be combined with algebraic effects in some way in functional programming. This. Algebraic effects seem very much destined for this purpose. You could safely run any code snippet (LLM generated or not) and know that it will only produce the effects you allowed.
      - nostrademons 1y ago
        Interesting. I hadn't heard of algebraic effects, but they remind me a lot of the Common Lisp condition system, or delimited continuations, or monads in Haskell. There's even a shoutout to monads in the top Google result for the context: https://overreacted.io/algebraic-effects-for-the-rest-of-us/ https://overreacted.io/algebraic-effects-for-the-rest-of-us/ I assume that the connection to capability security is that you use the effect handler to inject the capabilities you want to offer to the callee, and their access is limited to a particular dynamic scope, and is then revoked once it exits from that block? Handler types effectively provide for the callee and define what capabilities it may invoke, but the actual implementation is injected from a higher level?
        black_knight 1y ago
pdntspa 1y ago
How would that even work when the web is basically one big black box to the OS? Most of the stuff that matters to most consumers is on the web now anyway. I don't see how 'capabilities' would even work within the context of a user-agent LLM
- nostrademons 1y ago
  You'd have to rewrite most of the software used in modern life. Most of it is conceptually not built with a capability security model in mind. Instead of providing the LLM with access to your banking app, you need a new banking app that is built to provide access to your account and only your account, and additionally also offers a bunch of new controls like being able to set a budget for an operation and restrict the set of allowable payees to an allowlist. Instead of the app being "Log into Wells Fargo and send a payment with Zelle", the app becomes "Pay my babysitter no more than $200", and then the LLM is allowed to access that as part of its overall task scheduling. This is a major reason why capability security has failed in the marketplace.
dbmikus 1y ago
I'm going to be pedantic and note that iOS and Android both have the capability security model for their apps. And totally agree that instead of reinventing the wheel here, we should just lift from how operating systems work, for two reasons: 1. there's a bunch of work and proven systems there already 2. it uses tools that exist in training data, instead of net new tools
- nostrademons 1y ago
  App permissions in iOS and Android are both too coarse-grained to really be considered capabilities. Capabilities (at least as they exist in something like Eros or Capsicum) are more "You have access to this specific file" or "You can make connections to this specific server" rather than "You have access to files" and "You have access to the network". The file descriptor is passed in externally from a privileged process where the user explicitly decides what rights to give the process; there is no open() or connect() syscall available to user programs.
  - Terretta 1y ago
    One can sort of get there today combining something like attribute based access control, signed bearer tokens with attributes, and some sort of a degrees-of-delegability limiter that a bearer can pass along like a packet TTL. Did you want it in rust? - https://github.com/eclipse-biscuit/biscuit-rust https://github.com/eclipse-biscuit/biscuit-rust - https://www.biscuitsec.org/ https://www.biscuitsec.org/
  - saagarjha 1y ago
    This seems neat in theory but it is very difficult to actually do in practice. For example, let's say that you are allowed to make connections to a specific server. First, you have to get everyone onboard with isolating their code at that granularity, which requires a major rewrite that is easy to short-circuit by just being lazy and allowing overly broad permissions. But even then "a server" is hard to judge. Do you look at eTLD+1 (and ship PSL)? Do you look at IP (and find out that everything is actually Cloudflare)? Do you distinguish between an app that talks to the Gmail API, and one that is trying to reach Firebase for analytics? It's a hard problem. Most OSes do have some sort of capabilities as you've mentioned but the difficulty is not making them super fine-grained but actually designing them in a way that they have meaning.
oleszhulyn 1y ago
I'm building this for AI devices [1]. It's my YC F2025 submission. [1] https://uni-ai.com https://uni-ai.com

codethief 1y ago

> modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace. Fingers crossed that this is going to change now that there is increased demand due to AI workflows.

nostrademons 1y ago
I'm hoping, but not particularly optimistic. The dynamic that led to the Principle of Least Privilege failing in the market is that new technological innovations tend to succeed only when they enter new virgin territory that isn't already computerized, not when they're an incremental improvement over existing computer systems. And which markets will be successful tends to be very unpredictable. When you have those conditions, where new markets exist but are hard to find, the easiest way to expand into them is to let your software platforms do the greatest variety of things, and then expose that functionality to the widest array of developers possible in hopes that some of them will see a use you didn't think of. In other words, the opposite of the Principle of Least Privilege. This dynamic hasn't really changed with AI. If anything, it's accelerated. The AI boom kicked off when Sam Altman decided to just release ChatGPT to the general public without knowing exactly what it was for or building a fully-baked idea. There's going to be a lot of security misses in the process, some possibly catastrophic. IMHO the best shot that any capability-based software system has for success is to build out simplified versions of the most common consumer use-cases, and then wait for society to collapse. Because there's a fairly high likelihood of that, where the security vulnerabilities in existing software just allow a catastrophic compromise of the institutions of modern life, and a wholly new infrastructure becomes needed, and at that point you can point out exactly how we got this point and how to ensure it never happens again. On a small scale, there's historical precedence for this: a lot of the reason webapps took off in the early 2000s was because there was just a huge proliferation of worms and viruses targeting MS OSes in the late 90s and early 2000s, and it got to the point where consumers would only use webapps because they couldn't be confident that random software downloaded off the Internet wouldn't steal their credit card numbers.
- black_knight 1y ago
  Sometimes better ideas can be around for a really long time before they gain any mainstream traction. Some ideas which come to mind are anonymous functions and sum types with pattern matching, which are only recently finding their way into mainstream languages, despite having been around for ages. What it might take is a dedicated effort over time by a group of believers, to keep the ideas alive and create new attempts, new projects regularly. So that when there is a mainstream opening, there is the knownhow to implement them. I always include a lecture or two in my software security course (150 students per year), on capability based security. I am also on the lookout for projects which could use the ideas, but so far I have only vague ideas that they could be combined with algebraic effects in some way in functional programming.
  - codethief 1y ago
    > but so far I have only vague ideas that they could be combined with algebraic effects in some way in functional programming. This. Algebraic effects seem very much destined for this purpose. You could safely run any code snippet (LLM generated or not) and know that it will only produce the effects you allowed.
    - nostrademons 1y ago
      Interesting. I hadn't heard of algebraic effects, but they remind me a lot of the Common Lisp condition system, or delimited continuations, or monads in Haskell. There's even a shoutout to monads in the top Google result for the context: https://overreacted.io/algebraic-effects-for-the-rest-of-us/ https://overreacted.io/algebraic-effects-for-the-rest-of-us/ I assume that the connection to capability security is that you use the effect handler to inject the capabilities you want to offer to the callee, and their access is limited to a particular dynamic scope, and is then revoked once it exits from that block? Handler types effectively provide for the callee and define what capabilities it may invoke, but the actual implementation is injected from a higher level?
      - black_knight 1y ago

pdntspa 1y ago

How would that even work when the web is basically one big black box to the OS? Most of the stuff that matters to most consumers is on the web now anyway. I don't see how 'capabilities' would even work within the context of a user-agent LLM

nostrademons 1y ago
You'd have to rewrite most of the software used in modern life. Most of it is conceptually not built with a capability security model in mind. Instead of providing the LLM with access to your banking app, you need a new banking app that is built to provide access to your account and only your account, and additionally also offers a bunch of new controls like being able to set a budget for an operation and restrict the set of allowable payees to an allowlist. Instead of the app being "Log into Wells Fargo and send a payment with Zelle", the app becomes "Pay my babysitter no more than $200", and then the LLM is allowed to access that as part of its overall task scheduling. This is a major reason why capability security has failed in the marketplace.

dbmikus 1y ago

I'm going to be pedantic and note that iOS and Android both have the capability security model for their apps. And totally agree that instead of reinventing the wheel here, we should just lift from how operating systems work, for two reasons: 1. there's a bunch of work and proven systems there already 2. it uses tools that exist in training data, instead of net new tools

nostrademons 1y ago
App permissions in iOS and Android are both too coarse-grained to really be considered capabilities. Capabilities (at least as they exist in something like Eros or Capsicum) are more "You have access to this specific file" or "You can make connections to this specific server" rather than "You have access to files" and "You have access to the network". The file descriptor is passed in externally from a privileged process where the user explicitly decides what rights to give the process; there is no open() or connect() syscall available to user programs.
- Terretta 1y ago
  One can sort of get there today combining something like attribute based access control, signed bearer tokens with attributes, and some sort of a degrees-of-delegability limiter that a bearer can pass along like a packet TTL. Did you want it in rust? - https://github.com/eclipse-biscuit/biscuit-rust https://github.com/eclipse-biscuit/biscuit-rust - https://www.biscuitsec.org/ https://www.biscuitsec.org/
- saagarjha 1y ago
  This seems neat in theory but it is very difficult to actually do in practice. For example, let's say that you are allowed to make connections to a specific server. First, you have to get everyone onboard with isolating their code at that granularity, which requires a major rewrite that is easy to short-circuit by just being lazy and allowing overly broad permissions. But even then "a server" is hard to judge. Do you look at eTLD+1 (and ship PSL)? Do you look at IP (and find out that everything is actually Cloudflare)? Do you distinguish between an app that talks to the Gmail API, and one that is trying to reach Firebase for analytics? It's a hard problem. Most OSes do have some sort of capabilities as you've mentioned but the difficulty is not making them super fine-grained but actually designing them in a way that they have meaning.

oleszhulyn 1y ago

I'm building this for AI devices [1]. It's my YC F2025 submission. [1] https://uni-ai.com https://uni-ai.com