5 ms·

This..... this is the thing every architect attempting to roll containers out to production needs to read and re read Complexity at scale: Orchestration framew

by geggam 11y ago

This..... this is the thing every architect attempting to roll containers out to production needs to read and re read Complexity at scale: Orchestration frameworks (Rancher, MESOS/Aurora, Docker Swarm, LXD, OpenStack Containers, Kubernetes/Borg, etc) are only recently catching up to the container craze, there are too many competing models to list. Many have questionable or unaudited security or leave major requirements out, such as secret management. While containers may be easy to get working within a workstation or a few servers, scaling them up to production deployment is another challenge altogether, even assuming your application stack can be properly ``containerized''

dmpk2k 11y ago
My experience with Joyent's Triton stack has been both simple and positive. Much of the stuff in Section 10 either doesn't apply to the Triton stack (e.g. their implementation of the Docker server doesn't leave sockets lying around), or has already been solved there (e.g. their container implementation has already been hardened over a decade, and actually works well). I'm obviously quite the fan. This document is more an indictment of the current state of the Linux container ecosystem, rather than containers themselves. Don't conflate the two.
- geggam 11y ago
  Reading more and more of the document ... you might change your mind if you do so :)
  - dmpk2k 11y ago
    Yes, that's true. I was being a bit unfair.
kyrra 11y ago
The opinions stated here are my own, not necessarily those of Google. From my understanding Google was one of the main contributors of the core features that have allowed containers on Linux. Specifically cgroups and LXC. And they have have been running containers for 10 years: https://research.google.com/pubs/pub43438.html https://research.google.com/pubs/pub43438.html I'm not sure what you consider new, but it's definitely not super new to Google. I'm guessing your issue is that it's fairly new to being used in production across many companies, so security researchers are just starting to work on finding holes in it.
- jethro_tell 11y ago
  This is true but containers for mortals have lagged behind. Especially in management tools. There was cloudvz. Then cgmanage that couldn't run au pid 1 which has security concerns of its own. It wasn't until systemd that a small team could stand up containers with any type of scale. Basically everything before, more or less, took a proprietary config management setup.
  - ak217 11y ago
    We stood up containers, on very large scale, in a PaaS, with a very small team, in 2012 (long before systemd, or Docker for that matter, became a major force). This was made possible by the work done by Serge Hallyn, Stephane Graber, and others at Canonical. They delivered LXC and its integration with AppArmor, which in turn stood on top of the cgroups/namespaces work that Google contributed to the kernel. Ubuntu 12.04 was the release where the ability to securely run containers at scale became available to mere mortals.
- jsmthrowaway 11y ago
  You are correct. cgroups v1 came from Google (mostly Paul Menage, if I recall my history). OP is in a sense slightly correct without realizing it that Borg predates cgroups themselves, but I'd wager Googlers would agree that they're pretty happily married at this point. Kubernetes just doubles down on what Borg learned in some ways (and discards others). There's a distinction between containers the concept and Linux containers, of course, but yes. You've got it right.
- seabrookmx 11y ago
  Beyond that, there have been containers under Solaris and BSD jails for even longer. The concept is not new.
wstrange 11y ago
> secret management Kubernetes does have a reasonable solution here: http://kubernetes.io/docs/user-guide/secrets/ http://kubernetes.io/docs/user-guide/secrets/
peterwwillis 11y ago
This is something a security architect/analyst who hasn't looked into containers before should read, but it's not a great revelation on how to deploy containers or secure systems. Also, how is secret management a major requirement? That term doesn't even exist outside of open source Silicon Valley trendsters, btw. In the enterprise, only people with regulatory requirements use it, and then it's just called password management or credential management. More important requirements are typically the configuration of the services on the network, basic network security & access control, application security, and the security and training of internal users.
- geggam 11y ago
  Have fun trying to apply traditional models to the wonderful world of autoscaling containers utlizing service discovery :)
fpoling 11y ago
I do not see what is so difficult about management of secretes. Just add to the command/config file that starts container an option to bind mount file with secretes from the host like -v /path-on-the-host:/mount-point:ro with Docker and then use your favorite automation solution to transfer secrets. And I do not see how a generic container deployment solution can help with this as distribution of secrets are way too deployment/application specific.
- brown9-2 11y ago
  One issue with this strategy is that you can now only deploy that container/image on hosts with the file containing the image-specific secrets. If you want to have a large pool of generic hosts that can run any image, then you've just reduced the size of the pool for some images. If you choose instead to make every secret available to every host, then you've just spread your individual application secrets across many hosts.
  - fpoling 11y ago
    Typically secrets imply some per-container state that has to be transferred to the host in any case. Secretes can be transferred as a part of that container-specific state setup.

by geggam 11y ago

dmpk2k 11y ago

My experience with Joyent's Triton stack has been both simple and positive. Much of the stuff in Section 10 either doesn't apply to the Triton stack (e.g. their implementation of the Docker server doesn't leave sockets lying around), or has already been solved there (e.g. their container implementation has already been hardened over a decade, and actually works well). I'm obviously quite the fan. This document is more an indictment of the current state of the Linux container ecosystem, rather than containers themselves. Don't conflate the two.

geggam 11y ago
Reading more and more of the document ... you might change your mind if you do so :)
- dmpk2k 11y ago
  Yes, that's true. I was being a bit unfair.

kyrra 11y ago

The opinions stated here are my own, not necessarily those of Google. From my understanding Google was one of the main contributors of the core features that have allowed containers on Linux. Specifically cgroups and LXC. And they have have been running containers for 10 years: https://research.google.com/pubs/pub43438.html https://research.google.com/pubs/pub43438.html I'm not sure what you consider new, but it's definitely not super new to Google. I'm guessing your issue is that it's fairly new to being used in production across many companies, so security researchers are just starting to work on finding holes in it.

jethro_tell 11y ago
This is true but containers for mortals have lagged behind. Especially in management tools. There was cloudvz. Then cgmanage that couldn't run au pid 1 which has security concerns of its own. It wasn't until systemd that a small team could stand up containers with any type of scale. Basically everything before, more or less, took a proprietary config management setup.
- ak217 11y ago
  We stood up containers, on very large scale, in a PaaS, with a very small team, in 2012 (long before systemd, or Docker for that matter, became a major force). This was made possible by the work done by Serge Hallyn, Stephane Graber, and others at Canonical. They delivered LXC and its integration with AppArmor, which in turn stood on top of the cgroups/namespaces work that Google contributed to the kernel. Ubuntu 12.04 was the release where the ability to securely run containers at scale became available to mere mortals.
jsmthrowaway 11y ago
You are correct. cgroups v1 came from Google (mostly Paul Menage, if I recall my history). OP is in a sense slightly correct without realizing it that Borg predates cgroups themselves, but I'd wager Googlers would agree that they're pretty happily married at this point. Kubernetes just doubles down on what Borg learned in some ways (and discards others). There's a distinction between containers the concept and Linux containers, of course, but yes. You've got it right.
seabrookmx 11y ago
Beyond that, there have been containers under Solaris and BSD jails for even longer. The concept is not new.

wstrange 11y ago

> secret management Kubernetes does have a reasonable solution here: http://kubernetes.io/docs/user-guide/secrets/ http://kubernetes.io/docs/user-guide/secrets/

peterwwillis 11y ago

This is something a security architect/analyst who hasn't looked into containers before should read, but it's not a great revelation on how to deploy containers or secure systems. Also, how is secret management a major requirement? That term doesn't even exist outside of open source Silicon Valley trendsters, btw. In the enterprise, only people with regulatory requirements use it, and then it's just called password management or credential management. More important requirements are typically the configuration of the services on the network, basic network security & access control, application security, and the security and training of internal users.

geggam 11y ago
Have fun trying to apply traditional models to the wonderful world of autoscaling containers utlizing service discovery :)

fpoling 11y ago

I do not see what is so difficult about management of secretes. Just add to the command/config file that starts container an option to bind mount file with secretes from the host like -v /path-on-the-host:/mount-point:ro with Docker and then use your favorite automation solution to transfer secrets. And I do not see how a generic container deployment solution can help with this as distribution of secrets are way too deployment/application specific.

brown9-2 11y ago
One issue with this strategy is that you can now only deploy that container/image on hosts with the file containing the image-specific secrets. If you want to have a large pool of generic hosts that can run any image, then you've just reduced the size of the pool for some images. If you choose instead to make every secret available to every host, then you've just spread your individual application secrets across many hosts.
- fpoling 11y ago
  Typically secrets imply some per-container state that has to be transferred to the host in any case. Secretes can be transferred as a part of that container-specific state setup.