6 ms·

So the basic problem with *nix is its permission model. If we had truly separable security/privilege/resource domains then Linux wouldn't have needed container

So the basic problem with *nix is its permission model. If we had truly separable security/privilege/resource domains then Linux wouldn't have needed containers and simple processes and threads could have sufficed in place of Borg/docker/Kubernetes. There's a simpler and more powerful security model; capabilities. Capabilities fix 90% of the problems with *nix. There's currently no simple resource model. Everything is an ad-hoc human-driven heuristic for allocating resources to processes and threads, and is a really difficult problem to solve formally because it has to go beyond algorithmic complexity and care about the constant factors as well. The other *nix problem is "files". Files were a compromise between usability and precision but very few things are merely files. Devices and sockets sure aren't. There's a reason the 'file' utility exists; nothing is really just a file. Text files are actually text files + a context-free grammar (hopefully) and parser somewhere, or they're human-readable text (but probably with markup, so again a parser somewhere). Plenty of object models have come and gone; they haven't simplified computers (much less distributed computers), so we'll need some theory more powerful than anything we've had in the past to express relationships between computation, storage, networks, and identities.

dijit 5y ago
Containers never solved the permission model. They solved the packaging and idempotency problem. I really dislike when people assume containers give them security, it’s the wrong thing to think about. Containers allowed us to deploy reproducibly, that’s powerful.
- foxhill 5y ago
  > I really dislike when people assume containers give them security, it’s the wrong thing to think about. to be fair, there is lots of published text around suggesting that this _is_ the case. many junior to semi-experienced engineers i've known have at some point thought it's plausible to "ssh into" a container. they're seen as light-weight VMs, not as what they are - processes. > Containers allowed us to deploy reproducibly, that’s powerful. and it was done in the most "to bake an apple pie from scratch, you must first create the universe" approach.
  - mikeyjk 5y ago
    It is possible to do that though. I'm perhaps getting too caught up on 'plausible'.
  - phire 5y ago
    But you can ssh into a container. You just need to install sshd and launch it. You also need to create a user and set a password if you want to actually log in. Why? Because containers aren't a single process. It's a group of processes sharing a namespace. And you can totally use a container as a light-weight VM. While most containers have bash or a your application as pid 1, there is nothing stopping you launching a proper initrd as pid 1 and it will act much like a proper OS. Though, just because you can, doesn't mean you should.
    - otterley 5y ago
      I think you mean init, not initrd. An initrd is a RAM disk image loaded by Linux containing kernel file system and network drivers and is typically used to help minimize the size of the main kernel image.
- otabdeveloper4 5y ago
  Absolutely true. Docker replaced .tar.gz and .rpm, not chroots. Most of the time the chroot functionality of Docker is a hindrance, not a feature. We need chroots because we still haven't figured out packaging properly. (Maybe Nix will eventually solve this problem properly; some sort of docker-compose equivalent for managing systemd services is lacking at the moment.)
  - catern 5y ago
    Er, just as a historical note, one of the primary uses of chroots was for packaging. Just like how Docker does it. That, in a sense, was even the original motivation. The security usage of chroots was a later innovation.
- zajio1am 5y ago
  > Containers never solved the permission model. They solved the packaging and idempotency problem Disagree. Containers are primarily about separation and decoupling. Multiple services on one server often have plenty of ways to interact and see each other and are interdependent in non-trivial ways (e.g. if you want to upgrade the OS, you upgrade it for all services together). Services running each in its own container provides separation by default. OTOH, containers as a technology has nothing to do with packaging, reproducibility and deployment. Just these changes arrived together (e.g. with Docker) so they are often associated, but you can have e.g. LXC containers that can be managed in the same way as traditional servers (by ssh into a container).
  - dijit 5y ago
    LXC, freebsd jails & Solaris zones et al. are not the same as docker containers though. The former were built with security in mind. The latter was most assuredly not.
- cryptonector 5y ago
  I mean, containers can provide isolation. Linux has had a hard time getting that to be reliable because it started with the wrong model: building containers subtractively rather than additively. Though even starting with the right model, until you have isolation for every last bit of shared context that the OS provides (harder to identify than it may seem at first blush!) you won't have a complete solution. And yes, software-based containers will tend to have some leakage. Even sharing hardware with hardware isolation features might not be enough (hello row hammer). It would be good to have containers aim to provide the maximum possible isolation.
black_knight 5y ago
> There's a simpler and more powerful security model; capabilities. Capabilities fix 90% of the problems with *nix. What do you think about using filedescriptors as capabilties? Capsicum (for FreeBSD, I think) extends this notion quite a bit. Personally I feel it is not quite "right", but I haven't sat down and thought hard about what is missing. > we'll need some theory more powerful than anything we've had in the past to express relationships between computation, storage, networks, and identities. Do you have any particular things in mind which points in this direction? I would like to understand what the status quo is.
- benlivengood 5y ago
  I haven't looked at capsicum specifically, but from the simple overview I read it sounds like it is more similar to dropping root privileges when demonizing and not the basis for a whole-OS security model. E.g. there isn't (in my limited reading) a way to grant a new file descriptor to a process after it calls cap_enter. Consider a web browser that wants to download or upload a file; there should be a way for the operator to grant that permission to the browser from another process (the OS UI or similar) after it starts running. To be effective capabilities also need a way to be persistent so that a server daemon doesn't have to call cap_enter but can pick up its granted capabilities at startup. Capsicum looks like a useful way to build more secure daemons within Unix using a lot of capability features. I also think file descriptors are not the fundamental unit of capability. Capabilities should also cover processes, threads, and the objects managed by various other syscalls. > Do you have any particular things in mind which points in this direction? I would like to understand what the status quo is. Unfortunately I don't have great suggestions. The most secure model right now is seL4, and its capability model covers threads, message-passing endpoints, and memory allocation(subdivision) and retyping as kernel memory to create new capabilities and objects. The kernel is formally verified but afaik the application/user level is not fleshed out as a convenient development environment nor as a distributed computing environment. For distributed computing a capability model would have to satisfy and solve distributed trust issues which probably means capabilities based on cryptographic primitives, which for practical implementations would have to extend full trust between kernels in different machines for speed. But for universality it should be possible to work with capabilities at an abstraction level that allows both deep-trust distributed computers and more traditional single-machine trust domains without having to know or care which type of capabilities to choose when writing the software, only when running it. I think a foundation for universal capabilities needs support for different trust domains and a way to interoperate between them. 1. Identifying the controller for a particular capability, which trust domain it is in, and how to access it. 2. Converting capabilities between trust domains as the objects to which they refer move. 3. Managing any necessary identity/cryptographic tokens necessary to cross trust domains. 4. Controlling the ability to grant or use capabilities across trust domains. A simple example; a caller wants to invoke a capability on a utility process which produces an output, to which the caller wants to receive a capability to read the output. The processes may not live on the same machine. The processes may not be in the same trust domain. The resulting object may be on a third machine or trust domain. The caller may have inherited privacy enforcement on all owned capabilities that necessitates e.g. translating the binary code of the second process into a fully homomorphically encrypted circuit which can run on a different trust domain while preserving privacy and provisioning the necessary keys for this in the local trust domain so that the capability to the new object can actually read it. The process may migrate to a remote machine in a different trust domain in the middle of processing, in which case the OS needs to either fail the call (making for an unfortunately complicated distributed computer) or transparently snapshot or rollback the state of the process for migration, transmit it and any (potentially newly encrypted) data, and update the capabilities to reflect the new location and trust domain. Basically if the capability model isn't capable of solving these issues for what would be very simple local computing then it's never going to satisfy the OP's desire for a more simple distributed computation model. I think it's also clear why *nix is woefully short of being able to accomplish it. *\nix is inherently local and has a single trust domain and forces userland code to handle interaction with other trust domains except in the very limited model of network file systems (and in the case of NFS essentially an enforced single trust domain with synchronized user/group IDs)
cryptonector 5y ago
Windows has capabilities. It's the combination of handles (file, process, etc.) and access tokens. But you'll note no one is really deploying windows workloads to the cloud. Why? Well, because you'd still have to build a framework for managing all those permissions, and it hasn't been done. Also, you might end up with SVCHOST problem, where you host many different services/apps/whatever in one very threaded process because you can. Capabilities aren't necessarily simpler. Especially if you can delegate them without controls -- now you have no idea what the actual running permissions are, only the cold start baseline. No, I think the permissions thing is a red herring. Very much on the contrary, I think workload division into coarse-grained containers are great for permissions because fine-grained access control is hard to manage. Of course, you can't destroy complexity, only move it around, so if you should end up with many coarse-grained access control units then you'll still have a fine-grained access control system in the end. Files aren't really a problem either. You can add metadata to files on Linux using xattrs (I've built a custom HTTP server that takes some response headers for static resources, like Content-Type, from xattrs). The problem you're alluding to is duck-typing as opposed to static typing. Yes, it's a problem -- people are lazy, so they don't type-tag everything in highly lazy typing systems. So what? Windows also has this problem, just a bit less so than Unix. Python and JS are all the rage, and their type systems are lazy and obnoxious. It's not a problem with Unix. It's a problem with humans. Lack of discipline. Honestly, there are very few people who could use Haskell as a shell! > Plenty of object models have come and gone; Yeah, mostly because they suck. The right model is Haskell's (and related languages'). > so we'll need some theory more powerful than anything we've had in the past ... I think that's Haskell (which is still evolving) and its ecosystem (ditto). But at the end of the day, you'll still have very complex metadata to manage. What I don't understand is how all your points tie into Kubernetes being today's Multics. Kubernetes isn't motivated by Unix permissions sucking. We had fancy ACLs in ZFS in Solaris and still also ended up having Zones (containers). You can totally build an application-layer cryptographic capability system, running each app as its own isolated user/container, and to some degree this is happening with OAuth and such things, but that isn't what everyone is doing, all the time. Kubernetes is most definitely not motivated by Unix files being un-typed either. I hope readers end up floating the other, more on-topic top-level comments in this thread back to the top.

dijit 5y ago

Containers never solved the permission model. They solved the packaging and idempotency problem. I really dislike when people assume containers give them security, it’s the wrong thing to think about. Containers allowed us to deploy reproducibly, that’s powerful.

foxhill 5y ago
> I really dislike when people assume containers give them security, it’s the wrong thing to think about. to be fair, there is lots of published text around suggesting that this _is_ the case. many junior to semi-experienced engineers i've known have at some point thought it's plausible to "ssh into" a container. they're seen as light-weight VMs, not as what they are - processes. > Containers allowed us to deploy reproducibly, that’s powerful. and it was done in the most "to bake an apple pie from scratch, you must first create the universe" approach.
- mikeyjk 5y ago
  It is possible to do that though. I'm perhaps getting too caught up on 'plausible'.
- phire 5y ago
  But you can ssh into a container. You just need to install sshd and launch it. You also need to create a user and set a password if you want to actually log in. Why? Because containers aren't a single process. It's a group of processes sharing a namespace. And you can totally use a container as a light-weight VM. While most containers have bash or a your application as pid 1, there is nothing stopping you launching a proper initrd as pid 1 and it will act much like a proper OS. Though, just because you can, doesn't mean you should.
  - otterley 5y ago
    I think you mean init, not initrd. An initrd is a RAM disk image loaded by Linux containing kernel file system and network drivers and is typically used to help minimize the size of the main kernel image.
otabdeveloper4 5y ago
Absolutely true. Docker replaced .tar.gz and .rpm, not chroots. Most of the time the chroot functionality of Docker is a hindrance, not a feature. We need chroots because we still haven't figured out packaging properly. (Maybe Nix will eventually solve this problem properly; some sort of docker-compose equivalent for managing systemd services is lacking at the moment.)
- catern 5y ago
  Er, just as a historical note, one of the primary uses of chroots was for packaging. Just like how Docker does it. That, in a sense, was even the original motivation. The security usage of chroots was a later innovation.
zajio1am 5y ago
> Containers never solved the permission model. They solved the packaging and idempotency problem Disagree. Containers are primarily about separation and decoupling. Multiple services on one server often have plenty of ways to interact and see each other and are interdependent in non-trivial ways (e.g. if you want to upgrade the OS, you upgrade it for all services together). Services running each in its own container provides separation by default. OTOH, containers as a technology has nothing to do with packaging, reproducibility and deployment. Just these changes arrived together (e.g. with Docker) so they are often associated, but you can have e.g. LXC containers that can be managed in the same way as traditional servers (by ssh into a container).
- dijit 5y ago
  LXC, freebsd jails & Solaris zones et al. are not the same as docker containers though. The former were built with security in mind. The latter was most assuredly not.
cryptonector 5y ago
I mean, containers can provide isolation. Linux has had a hard time getting that to be reliable because it started with the wrong model: building containers subtractively rather than additively. Though even starting with the right model, until you have isolation for every last bit of shared context that the OS provides (harder to identify than it may seem at first blush!) you won't have a complete solution. And yes, software-based containers will tend to have some leakage. Even sharing hardware with hardware isolation features might not be enough (hello row hammer). It would be good to have containers aim to provide the maximum possible isolation.

black_knight 5y ago

> There's a simpler and more powerful security model; capabilities. Capabilities fix 90% of the problems with *nix. What do you think about using filedescriptors as capabilties? Capsicum (for FreeBSD, I think) extends this notion quite a bit. Personally I feel it is not quite "right", but I haven't sat down and thought hard about what is missing. > we'll need some theory more powerful than anything we've had in the past to express relationships between computation, storage, networks, and identities. Do you have any particular things in mind which points in this direction? I would like to understand what the status quo is.

benlivengood 5y ago
I haven't looked at capsicum specifically, but from the simple overview I read it sounds like it is more similar to dropping root privileges when demonizing and not the basis for a whole-OS security model. E.g. there isn't (in my limited reading) a way to grant a new file descriptor to a process after it calls cap_enter. Consider a web browser that wants to download or upload a file; there should be a way for the operator to grant that permission to the browser from another process (the OS UI or similar) after it starts running. To be effective capabilities also need a way to be persistent so that a server daemon doesn't have to call cap_enter but can pick up its granted capabilities at startup. Capsicum looks like a useful way to build more secure daemons within Unix using a lot of capability features. I also think file descriptors are not the fundamental unit of capability. Capabilities should also cover processes, threads, and the objects managed by various other syscalls. > Do you have any particular things in mind which points in this direction? I would like to understand what the status quo is. Unfortunately I don't have great suggestions. The most secure model right now is seL4, and its capability model covers threads, message-passing endpoints, and memory allocation(subdivision) and retyping as kernel memory to create new capabilities and objects. The kernel is formally verified but afaik the application/user level is not fleshed out as a convenient development environment nor as a distributed computing environment. For distributed computing a capability model would have to satisfy and solve distributed trust issues which probably means capabilities based on cryptographic primitives, which for practical implementations would have to extend full trust between kernels in different machines for speed. But for universality it should be possible to work with capabilities at an abstraction level that allows both deep-trust distributed computers and more traditional single-machine trust domains without having to know or care which type of capabilities to choose when writing the software, only when running it. I think a foundation for universal capabilities needs support for different trust domains and a way to interoperate between them. 1. Identifying the controller for a particular capability, which trust domain it is in, and how to access it. 2. Converting capabilities between trust domains as the objects to which they refer move. 3. Managing any necessary identity/cryptographic tokens necessary to cross trust domains. 4. Controlling the ability to grant or use capabilities across trust domains. A simple example; a caller wants to invoke a capability on a utility process which produces an output, to which the caller wants to receive a capability to read the output. The processes may not live on the same machine. The processes may not be in the same trust domain. The resulting object may be on a third machine or trust domain. The caller may have inherited privacy enforcement on all owned capabilities that necessitates e.g. translating the binary code of the second process into a fully homomorphically encrypted circuit which can run on a different trust domain while preserving privacy and provisioning the necessary keys for this in the local trust domain so that the capability to the new object can actually read it. The process may migrate to a remote machine in a different trust domain in the middle of processing, in which case the OS needs to either fail the call (making for an unfortunately complicated distributed computer) or transparently snapshot or rollback the state of the process for migration, transmit it and any (potentially newly encrypted) data, and update the capabilities to reflect the new location and trust domain. Basically if the capability model isn't capable of solving these issues for what would be very simple local computing then it's never going to satisfy the OP's desire for a more simple distributed computation model. I think it's also clear why *nix is woefully short of being able to accomplish it. *\nix is inherently local and has a single trust domain and forces userland code to handle interaction with other trust domains except in the very limited model of network file systems (and in the case of NFS essentially an enforced single trust domain with synchronized user/group IDs)

cryptonector 5y ago

Windows has capabilities. It's the combination of handles (file, process, etc.) and access tokens. But you'll note no one is really deploying windows workloads to the cloud. Why? Well, because you'd still have to build a framework for managing all those permissions, and it hasn't been done. Also, you might end up with SVCHOST problem, where you host many different services/apps/whatever in one very threaded process because you can. Capabilities aren't necessarily simpler. Especially if you can delegate them without controls -- now you have no idea what the actual running permissions are, only the cold start baseline. No, I think the permissions thing is a red herring. Very much on the contrary, I think workload division into coarse-grained containers are great for permissions because fine-grained access control is hard to manage. Of course, you can't destroy complexity, only move it around, so if you should end up with many coarse-grained access control units then you'll still have a fine-grained access control system in the end. Files aren't really a problem either. You can add metadata to files on Linux using xattrs (I've built a custom HTTP server that takes some response headers for static resources, like Content-Type, from xattrs). The problem you're alluding to is duck-typing as opposed to static typing. Yes, it's a problem -- people are lazy, so they don't type-tag everything in highly lazy typing systems. So what? Windows also has this problem, just a bit less so than Unix. Python and JS are all the rage, and their type systems are lazy and obnoxious. It's not a problem with Unix. It's a problem with humans. Lack of discipline. Honestly, there are very few people who could use Haskell as a shell! > Plenty of object models have come and gone; Yeah, mostly because they suck. The right model is Haskell's (and related languages'). > so we'll need some theory more powerful than anything we've had in the past ... I think that's Haskell (which is still evolving) and its ecosystem (ditto). But at the end of the day, you'll still have very complex metadata to manage. What I don't understand is how all your points tie into Kubernetes being today's Multics. Kubernetes isn't motivated by Unix permissions sucking. We had fancy ACLs in ZFS in Solaris and still also ended up having Zones (containers). You can totally build an application-layer cryptographic capability system, running each app as its own isolated user/container, and to some degree this is happening with OAuth and such things, but that isn't what everyone is doing, all the time. Kubernetes is most definitely not motivated by Unix files being un-typed either. I hope readers end up floating the other, more on-topic top-level comments in this thread back to the top.