18 ms·

Introducing Docker Secrets Management

264 points by ferrantim 10y ago

diogomonicapt 10y ago
I think with this release things have come full circle for me. I was part of the team that 5 years ago built Keywhiz at Square, starting the whole "secrets should be files exposed as an in-memory filesystem" thing. Building it a second time was interesting. One of the biggest reasons why Keywhiz didn't go anywhere was the fact that it is incredibly hard to setup, and requires you to bring your own PKI. This time we didn't make that mistake and integrated directly into Swarm, which is the right place for it to live, and turns setting up your own PKI into a one-liner. Anyway, AMA. Disclosure: I work on the Docker Security team
- tdhz77 10y ago
  Can you explain to me the benefits of this like I'm 5 years old? Thanks, and sorry I have technical debt.
  - LittleFuckinCat 10y ago
    Probably they were concerned that users have to store the secrets (certificates, etc...) in the container filesystem - or mount/share the location on the host were the certificate for a specific app is present. Probably there are more ways to do it, but also probably none is end-to-end secure. With this you can securely store the certificates; takes away the developer's responsibility of taking care of managing and distributing them to their apps, swarm will take care of that in a secure end-to-end fashion.
  - LeanderK 10y ago
    Since diogomonicapt didn't answer i'll try to give an ELI5 answer ;) Docker-containers have many advantages, but one of the big drawbacks until now was that it was really hard to pass secrets into them. Imagine you are a simple, stupid Java-container. You handle HTTP-Request and need to access other services like Databases in order to function. In order to connect to the postgres-instance you need to know 3 things: the URL, the database-name and a password. Getting the URL and the database name is easy, just pass them via environment-variables. The tricky thing is how to pass the secret. Environment variables are not encrypted and super easy to read as soon as you have gained access to the container. There were some solutions, but they were complicated. So most just passed them via env-variables. Huge security risk. Very bad. This solves the problem.
    - diogomonicapt 10y ago
      Way better than I could have answered ;)
    - vbernat 10y ago
      If the environment is super easy to read when you have gained access to the container, so is a filesystem... The use of secrets is more to transmit secrets to the container without giving it to people.
      - diogomonicapt 10y ago
        The problem is applications unintentionally leaking the ENV. Think a hoptoad exception that attaches the current ENV to the report that sends up to the remote server. Or think about you exec'ing imagemagik and now the process running potentially adversarial code also has access to your parent's env. Or think about an application crashing and doing an unintentional core dump to disk.
- avaid1996 10y ago
  Looks awesome, congratulations on the release!
- koolba 10y ago
  Are the values loaded on the fly each time they're read or is the /run/secrets mount statically defined when the container starts up? If it's static, follow up to that would be, how are changes propagated to already running instances? Is there a way to restrict access to reading said secrets once the container is running? Say you're running possibly malicious code that has access to the local filesystem (ex: CI test runner) is there a way to restrict a process from reading those files? Can we simply delete them (i.e. "burn after reading") or are they fully virtual?
  - sly010 10y ago
    > Is there a way to restrict access to reading said secrets once the container is running? I don't know anything about docker, but the best way I found to do this in linux in general was aa_changehat() [0]. You write an apparmor profile for startup and a sub-profile for the running app/service. After setup, you call aa_changehat() to switch the current process to use the subprofile. You then throw away your magic token, so there is no way to switch back. You don't even have to link to libapparmor, under the hood aa_changehat() just writes some string somewhere in /proc, so you can replicate that. Note, I haven't actually done this, but working on it right now. [0] http://manpages.ubuntu.com/manpages/wily/man2/aa_change_hat.2.html http://manpages.ubuntu.com/manpages/wily/man2/aa_change_hat....
- eicnix 10y ago
  From what I understand there are no secruity mechanisms other than authentication to the cluster. How do you prevent somebody that got access to the cluster to just read all your secrets? Can I plug in my own secret store?
  - sarnowski 10y ago
    That is the same trust boundary as in Kubernetes currently: https://kubernetes.io/docs/user-guide/secrets/ https://kubernetes.io/docs/user-guide/secrets/ It is the most important step that you can package containers without having to know the production secrets and to have a "standard" was to retrieve them. No one, without production access, has a way to obtain them.
    - cpuguy83 10y ago
      Note exactly the same (for now): > Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet > If multiple replicas of etcd are run, then the secrets will be shared between them. By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.
      - smarterclayton 10y ago
        Agree, it would be better if the nodes could not trivially request other secrets. But you can request new containers placed on any node and a root escape that works on one node almost certainly works on the rest. We usually recommend subdividing the node acls by namespace when running disjoint node sets (where tenant A can't schedule onto tenant B's nodes). More fiddly than it has to be in Kubernetes today.
        piva00 10y ago
        We are currently facing the same problem and came to the same conclusion, we are subdividing our node pools by namespace (and using namespacing for multi-tenancy) but it "feels" very convoluted and a smell that there's a feature missing in Kubernetes itself to handle this more graciously.
- mihular 10y ago
  What about non swarm containers? Like straight docker and docker-compose?
  - odc 10y ago
    docker-compose 1.11.0 will be released soon with support for secrets, probably using the same system.
    - diogomonicapt 10y ago
      Exactly. The new version of compose supports defining docker secrets inside of the compose file.
      - mihular 10y ago
        But the question is whether secrets can be used outside swarm. It doesn't look so at first glance.
  - nicolaka 10y ago
    secrets only work with Swarm services. New compose (1.11) uses a new format version ( v3.1) that allows you to add secrets for services and deploy using docker client ( no need for docker-compose). I don't think there's any plans atm to support secrets for non-swarm containers :(
- mreithub 10y ago
  So I guess the best-practice approach would be to chown/chmod/rm the file after reading its contents (assuming it'll be restored when you restart the container). So far my approach was defining environment variables in the various docker-compoose files (in a separate deployment git repo), but this looks like a really nice alternative. Do you have plans to update the library images to give us a choice between using ENV and secrets (for DB server passwords and the like)? On an aside: I've gotten the Docker Datacenter announcement mail today. I only took the time to skim its contents quickly and at first thought this was a DDC-only thing. Glad to hear it isn't, keep up the awesome work. edit: clarified my docker-compose usage
  - diogomonicapt 10y ago
    Thanks! - Exposing secrets as in-memory files has a lot of advantages over ENV variables (harder to leak). - We already started updating a few images (MySQL, for example), so they can use Docker secrets. - Definitely not DDC only, but note that RBAC over secrets is a feature of the commercial product.
    - bluebeard 10y ago
      Thanks, I'm interested in trying this out.
- neomantra 10y ago
  This secrets infrastructure is only available at run-time and with Swarm? Are you looking to handle build-time secrets?
  - diogomonicapt 10y ago
    Yes: https://github.com/docker/docker/pull/30637 https://github.com/docker/docker/pull/30637
    - neomantra 10y ago
      OMG, so much smoother than what I do now. Thanks for the link.
- himanshuy 10y ago
  Is Docker PCI compliant? My application deals with a lot of PII data. There was a lot of concern in my team over SOC-2 and PCI compliance when I suggested using Docker. We also talked to one of the solution architects from Amazon and he wasn't sure if Docker is ready for PII data.
  - diogomonicapt 10y ago
    You can make a PCI compliant installation w/ Docker yes.
  - sarnowski 10y ago
    A single software piece can't be PCI compliant but only how you use it. You have to argue that Docker uses the Linux isolation mechanisms that make those containers virtual machines in the sense and spirit of PCI. Treating containers as VMs makes some other requirements even easier like the request to have a minimal system and to only have one function per server - thats how you want containers to work anyway. (Btw PCI has nothing to do with PII.)
  - Diederich 10y ago
    > solution architects from Amazon and he wasn't sure if Docker is ready for PII data. I'm curious if you can relate what his reasoning was? Docker is, among other things, a wrapper around a bunch of Linux kernel functions, the likes of which have been used for many, many years by companies like Google to facilitate all kinds of useful isolation.
    - paulddraper 10y ago
      I think it's pretty well agreed that multi-tenant Docker isn't a good security solution. Use it for for operational isolation, not for critical security isolation.
      - Diederich 10y ago
        That's what I've heard as well, and I agree with that assessment, given that the requirement is to separate untrusted code execution. At the same time, going back to the initial quote: > solution architects from Amazon and he wasn't sure if Docker is ready for PII data A lot of companies are using containers to execute code that manages all kinds of regulated data right now, ya? The security limitations of Linux namespaces and friends are mostly related to the execution of untrusted code.
- mikegerwitz 10y ago
  The docs explicitly state "You cannot remove a secret that a running service is using". I'm not comfortable with sensitive secrets remaining plaintext in memory after the running service possibly does not need it anymore (e.g. maybe the service only needs it on startup). It seems like you could rotate the key with something else, but that doesn't seem ideal. Or can the container itself remove the secret "file"? Consequently, because it's mounted as a filesytem, what if the service is compromised and vulnerable to arbitrary code execution, directory traversal, etc? The secret could then be leaked. Am I misinterpreting something? How would others here handle this? Edit: To clarify: rotating a secret will cause the service to restart. So I guess by "doesn't seem ideal", I mean it doesn't seem like an option.
  - cpuguy83 10y ago
    You can update a service to not use a secret once it's not needed. Of course this will restart the task(s) I think. Definitely worth thinking about. Maybe a single-read secret or something. If a service is compromised, you should always assume the secret is compromised.
    - mikegerwitz 10y ago
      > Maybe a single-read secret or something. I like this idea. As a benefit, you wouldn't have to rely on developers of individual services to make sure the secret is cleared. This would also have the benefit of causing an error if something else happened to read the secret before the intended service, indicating a possible compromise (or maybe just a misconfiguration). (I use a similar concept with `xclip -l' to allow reading passwords from the clipboard a single time before it quits.)
- sandGorgon 10y ago
  is anyone running this stuff in production yet ? not just secrets.. but the whole Docker Swarm with the v3 compose file format thing ? We have been testing out production clusters with Swarm 1.13 and I think its neat. But what do I know - I'm not a cloud hosting specialist. And the lack of success stories out there is making us nervous.
  - kossae 10y ago
    I can honestly say the only thing we found the full Docker line up (Containers, Swarm, Machine, etc.) good for large scale dev or staging/testing stuff. There are definitely uses for each of these tools, or other combinations, that have been successful for other people. It seems like Docker has a lot to focus on keeping up with the superior orchestration platforms that are out now. This in turn means quickly outdated (or missing) docs, inconsistent behavior of certain features, and other little "gotchas". Depending on the size of your team, I would recommend Kontena (www.kontena.io) or Kubernetes (https://kubernetes.io/ https://kubernetes.io/). While I haven't used the latter, a group of two devs is re-architecting our entire system to use Docker and Kontena. Nearing the end, the whole process seemed incredibly fast to iterate. While K8s seems far more proven in real production environments, it seemed much more daunting to us with more features than we needed for our use case.
    - raesene9 10y ago
      I'd recommend people read the "Risks" section of Kubernetes Docs on secrets before using it... https://kubernetes.io/docs/user-guide/secrets/#risks https://kubernetes.io/docs/user-guide/secrets/#risks. secrets stored in plain text and (by default) transmitted in plain text between etcd services, does not fill me with confidence on their security.
      - cpitman 10y ago
        Both of these are easy to fix. Configure etcd with peer certificates for clustering and a cert for client-server connections (https://coreos.com/etcd/docs/latest/v2/security.html https://coreos.com/etcd/docs/latest/v2/security.html). If you need encryption at rest, encrypt the filesystem. If setting up a secure cluster is daunting, then use a distribution that handles it for you. OpenShift (https://www.openshift.org/ https://www.openshift.org/) is built on kubernetes, and it's install is secure by default. Disclaimer: I work for Red Hat, and spend lots of time on OpenShift consulting.
- m_mueller 10y ago
  This is a bit off-topic, but I'll hope for the second "A" in AMA: I still have a bit trouble figuring out how docker envisions URL routing to work. Currently we use a simple setup with Hipache that forwards to our hosts running containers und multiple ports. This would work for failover and scaling as well, however at the point where DBs come into play it gets troublesome because hipache just routes in a round-robin manner instead of keeping routes persistent for each client IP (so you could end up showing them different replicas that are not in sync yet). Anyways, my issue is: Does Swarm integrate a solution for URL routing and if so, does it tackle the persistency problem?
  - nicolaka 10y ago
    In the commercial product(DDC) there is something called HTTP Routing Mesh that basically adds URL routing to backend services using the underlying routing mesh in Swarm. It supports both HTTP and HTTPS ( w/SNI). It adds RBAC too to make sure only permitted services can be exposed externally. More details here: https://success.docker.com/Datacenter/Apply/Docker_Reference_Architecture%3A_Universal_Control_Plane_2.0_Service_Discovery_and_Load_Balancing#The_HTTP_Routing_Mesh https://success.docker.com/Datacenter/Apply/Docker_Reference... Hope this helps!
unpythonic 10y ago
This looks amazing. Solving secret distribution across containers will be very useful. Being able to see via `docker secret` exactly which services are using which secrets is an unexpected treat. I'm a little worried about two aspects of what has been shown. From the article it shows: $ docker exec $(docker ps --filter name=redis -q) ls -l /run/secrets total 4 -r--r--r-- 1 root root 17 Dec 13 22:48 my_secret_data From this we can tell exactly how long the secret is. If the secret service didn't do it for us, I'd like for the secrets to be null-padded to a uniform 2-4K of bytes. I'm also a bit worried that the default protection on the file has it set to world-readable. Since it appears that secret distribution is independent of the container setup itself, there doesn't appear to be any way of setting ownership and permissions on this file. That is, if one were able to chmod/chown the file in the Dockerfile, running a `docker service update --secret-rm` and `docker service update --secret-add` would reset such 'fixes'. A great start, and I can't wait to start using it.
- cyli 10y ago
  You can specify the UID, GID, and mode of the secret from within the container: https://docs.docker.com/engine/reference/commandline/service_create/#/create-a-service-with-secrets https://docs.docker.com/engine/reference/commandline/service..., and pass the same configuration to the service update command as well.
  - unpythonic 10y ago
    Oh, that's great to know. That coupled with filling out the size to an uninteresting length with nulls (or another delimiter) covers my concerns. For example: $ echo "my secret" | dd bs=512 conv=sync | docker secret create my_secret_data -
simplehuman 10y ago
This is really nice. What is the equivalent in the kubernetes world?
- eicnix 10y ago
  Kubernetes supported secrets for nearly 2 years: https://kubernetes.io/docs/user-guide/secrets/ https://kubernetes.io/docs/user-guide/secrets/ They currently debate over pluggable secret stores: https://github.com/kubernetes/kubernetes/issues/10439 https://github.com/kubernetes/kubernetes/issues/10439 You can use Hashicorp Vault as a third party resource to store your secrets: https://github.com/Boostport/kubernetes-vault https://github.com/Boostport/kubernetes-vault Kubernetes also supports RBAC to control access to secrets: https://kubernetes.io/docs/admin/authorization/ https://kubernetes.io/docs/admin/authorization/
  - tonyhb 10y ago
    Secrets as in passwords and keys stored at rest in plain text. So not really secrets.
    - benth 10y ago
      It's not so much the plain text part that bothers me, it's the access control. Quoting the docs at https://kubernetes.io/docs/user-guide/secrets/#security-properties https://kubernetes.io/docs/user-guide/secrets/#security-prop...: "Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet. It is a planned feature to only send secrets to nodes that actually require them, to restrict the impact of a root exploit on a single node." As your cluster grows, your risk grows.
tlrobinson 10y ago
It would be neat if you could inject secrets as environment variables. Something like: --secret="my_secret:MY_SECRET" Obviously you could have a script that does MY_SECRET=$(cat /run/secrets/my_secret) ./whatever but a lot of existing containers can be configured via environment variables.
- diogomonicapt 10y ago
  We explicitly chose not to support secrets as ENV variables, since they are prone to being leaked (child processes inhering parent's env; easy to leak ps -e; bug reports usually include ENV of the application; core dumps include ENV of the application, etc etc)
  - tlrobinson 10y ago
    Makes sense, but how do child processes not have access to the secrets files?
    - paulddraper 10y ago
      They do. That point is less valid.
    - diogomonicapt 10y ago
      My unstated assumption was that when calling something like imagetragick you would be doing the appropriate privilege dropping using setuid and setgid. This is obviously not necessarily the case, but it's at least a lot more common than people cleaning up the ENV before creating a child process.
doublerebel 10y ago
Comparing this to Vault (with which I have been very satisfied): The article mentions a single master key for cluster encryption. Are there any plans to split this as in Shamir's Secret Sharing? How do secrets and secret authorizations renew and expire? Any plans for limited-use tokens/secrets?
heroprotagonist 10y ago
This is interesting, but in my opinion, it's not quite as universally useful as some of the secret management in other security tools because you have to explicitly manage the secret within your application by reading that in-memory filesystem. I'd much prefer passing a secret as an environment variable if that can be done securely. It's possible with some tools, but not out of the box with Docker itself. eg, with one tool, you can do: # docker run --rm -e PASSWORD={supersecret_password} someimage:latest program Then '{supersecret_password}' gets replaced in the container at runtime with the value stored in the tool or from an integration with a separate dedicated secret management tool like HashiCorp Vault, and value gets masked external to the container such as when running 'docker inspect' command. The benefit is that you don't need to modify or maintain a lot of pre-packaged applications that read environment variables instead of looking to the contents of a file on the disk, so it just works out the box. However, under ordinary circumstances, you may not want to pass secrets as environment variables in docker (or at least be careful about it). A 'docker inspect' command can show any docker user the environment variable and its value, if you don't have a tool to encrypt the contents.
- eicnix 10y ago
  Kubernetes supports this already. You have the choice to either mount the secret into the container file system or set it as a environment variable. https://kubernetes.io/docs/user-guide/secrets/#using-secrets-as-environment-variables https://kubernetes.io/docs/user-guide/secrets/#using-secrets...
  - tonyhb 10y ago
    But Kubernetes secrets are stored - and by default transmitted amongst managers - in plain text. And Kube will share secrets with all kubelets, no questions asked. They're not "secrets".
    - eicnix 10y ago
      Yes, the kubernetes secrets resource is not a replacement for a secret store. Your specific point is being worked on: https://github.com/kubernetes/kubernetes/issues/12742 https://github.com/kubernetes/kubernetes/issues/12742 You can use Vault(https://github.com/Boostport/kubernetes-vault https://github.com/Boostport/kubernetes-vault) or extend Kubernetes easily for your specific secret as a replacement if you need extra security.
daveguy 10y ago
Not sure if anyone from docker is watching, but the images are cut off at the side on mobile.
- ManoMarks 10y ago
  Yes, someone from Docker is watching :-). I've alerted the blog team. Thanks for letting us know.
deleted 10y ago
[deleted]
justincormack 10y ago
Comparison of this with other secretes management frameworks here https://medium.com/on-docker/secrets-and-lie-abilities-the-state-of-modern-secret-management-2017-c82ec9136a3d#.ur0pnuirw https://medium.com/on-docker/secrets-and-lie-abilities-the-s...
- tonyhb 10y ago
  wrote this as a comment when that article was posted to YC: tl;dr: author looks at secret management services and reviews them KeyWhiz: Provides everything you need but with complex PKI management, meaning setup and maintenance is a pain. Secure. Vault: A+ would test again. Awesome rotation policies, on-demand secret generation via backends, master key sharing. Legit and secure, everything you need but has to be configured on top of your cluster. Docker: Super easy to use, and it's built in. 10/10 would use again. Keys encrypted at rest, keys encrypted over the wire, and shared with only nodes who need them. Secure all round Kube: Totally insecure. Plaintext at rest, plaintext over the network, shared everywhere. Basically a plaintext POC
  - Leon 10y ago
    Vault is really wonderful to work with, and its integration with consul makes it phenomenally powerful.
    - jsfitzsimmons 10y ago
      What kind of stuff do you get from the integration? I mean, aside from the obvious stuff like a clustered storage backend and leader election?
lyschoening 10y ago
Will we be able to use this on Docker Cloud soon?
- justincormack 10y ago
  I believe it is coming soon yes.
mnm1 10y ago
Is there a way to manage secrets during the build process itself without specifying them on the command line / docker compose files?

264 points by ferrantim 10y ago

I think with this release things have come full circle for me. I was part of the team that 5 years ago built Keywhiz at Square, starting the whole "secrets should be files exposed as an in-memory filesystem" thing. Building it a second time was interesting. One of the biggest reasons why Keywhiz didn't go anywhere was the fact that it is incredibly hard to setup, and requires you to bring your own PKI. This time we didn't make that mistake and integrated directly into Swarm, which is the right place for it to live, and turns setting up your own PKI into a one-liner. Anyway, AMA. Disclosure: I work on the Docker Security team

tdhz77 10y ago
Can you explain to me the benefits of this like I'm 5 years old? Thanks, and sorry I have technical debt.
- LittleFuckinCat 10y ago
  Probably they were concerned that users have to store the secrets (certificates, etc...) in the container filesystem - or mount/share the location on the host were the certificate for a specific app is present. Probably there are more ways to do it, but also probably none is end-to-end secure. With this you can securely store the certificates; takes away the developer's responsibility of taking care of managing and distributing them to their apps, swarm will take care of that in a secure end-to-end fashion.
- LeanderK 10y ago
  Since diogomonicapt didn't answer i'll try to give an ELI5 answer ;) Docker-containers have many advantages, but one of the big drawbacks until now was that it was really hard to pass secrets into them. Imagine you are a simple, stupid Java-container. You handle HTTP-Request and need to access other services like Databases in order to function. In order to connect to the postgres-instance you need to know 3 things: the URL, the database-name and a password. Getting the URL and the database name is easy, just pass them via environment-variables. The tricky thing is how to pass the secret. Environment variables are not encrypted and super easy to read as soon as you have gained access to the container. There were some solutions, but they were complicated. So most just passed them via env-variables. Huge security risk. Very bad. This solves the problem.
  - diogomonicapt 10y ago
    Way better than I could have answered ;)
  - vbernat 10y ago
    If the environment is super easy to read when you have gained access to the container, so is a filesystem... The use of secrets is more to transmit secrets to the container without giving it to people.
    - diogomonicapt 10y ago
      The problem is applications unintentionally leaking the ENV. Think a hoptoad exception that attaches the current ENV to the report that sends up to the remote server. Or think about you exec'ing imagemagik and now the process running potentially adversarial code also has access to your parent's env. Or think about an application crashing and doing an unintentional core dump to disk.
avaid1996 10y ago
Looks awesome, congratulations on the release!
koolba 10y ago
Are the values loaded on the fly each time they're read or is the /run/secrets mount statically defined when the container starts up? If it's static, follow up to that would be, how are changes propagated to already running instances? Is there a way to restrict access to reading said secrets once the container is running? Say you're running possibly malicious code that has access to the local filesystem (ex: CI test runner) is there a way to restrict a process from reading those files? Can we simply delete them (i.e. "burn after reading") or are they fully virtual?
- sly010 10y ago
  > Is there a way to restrict access to reading said secrets once the container is running? I don't know anything about docker, but the best way I found to do this in linux in general was aa_changehat() [0]. You write an apparmor profile for startup and a sub-profile for the running app/service. After setup, you call aa_changehat() to switch the current process to use the subprofile. You then throw away your magic token, so there is no way to switch back. You don't even have to link to libapparmor, under the hood aa_changehat() just writes some string somewhere in /proc, so you can replicate that. Note, I haven't actually done this, but working on it right now. [0] http://manpages.ubuntu.com/manpages/wily/man2/aa_change_hat.2.html http://manpages.ubuntu.com/manpages/wily/man2/aa_change_hat....
eicnix 10y ago
From what I understand there are no secruity mechanisms other than authentication to the cluster. How do you prevent somebody that got access to the cluster to just read all your secrets? Can I plug in my own secret store?
- sarnowski 10y ago
  That is the same trust boundary as in Kubernetes currently: https://kubernetes.io/docs/user-guide/secrets/ https://kubernetes.io/docs/user-guide/secrets/ It is the most important step that you can package containers without having to know the production secrets and to have a "standard" was to retrieve them. No one, without production access, has a way to obtain them.
  - cpuguy83 10y ago
    Note exactly the same (for now): > Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet > If multiple replicas of etcd are run, then the secrets will be shared between them. By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.
    - smarterclayton 10y ago
      Agree, it would be better if the nodes could not trivially request other secrets. But you can request new containers placed on any node and a root escape that works on one node almost certainly works on the rest. We usually recommend subdividing the node acls by namespace when running disjoint node sets (where tenant A can't schedule onto tenant B's nodes). More fiddly than it has to be in Kubernetes today.
      - piva00 10y ago
        We are currently facing the same problem and came to the same conclusion, we are subdividing our node pools by namespace (and using namespacing for multi-tenancy) but it "feels" very convoluted and a smell that there's a feature missing in Kubernetes itself to handle this more graciously.
mihular 10y ago
What about non swarm containers? Like straight docker and docker-compose?
- odc 10y ago
  docker-compose 1.11.0 will be released soon with support for secrets, probably using the same system.
  - diogomonicapt 10y ago
    Exactly. The new version of compose supports defining docker secrets inside of the compose file.
    - mihular 10y ago
      But the question is whether secrets can be used outside swarm. It doesn't look so at first glance.
- nicolaka 10y ago
  secrets only work with Swarm services. New compose (1.11) uses a new format version ( v3.1) that allows you to add secrets for services and deploy using docker client ( no need for docker-compose). I don't think there's any plans atm to support secrets for non-swarm containers :(
mreithub 10y ago
So I guess the best-practice approach would be to chown/chmod/rm the file after reading its contents (assuming it'll be restored when you restart the container). So far my approach was defining environment variables in the various docker-compoose files (in a separate deployment git repo), but this looks like a really nice alternative. Do you have plans to update the library images to give us a choice between using ENV and secrets (for DB server passwords and the like)? On an aside: I've gotten the Docker Datacenter announcement mail today. I only took the time to skim its contents quickly and at first thought this was a DDC-only thing. Glad to hear it isn't, keep up the awesome work. edit: clarified my docker-compose usage
- diogomonicapt 10y ago
  Thanks! - Exposing secrets as in-memory files has a lot of advantages over ENV variables (harder to leak). - We already started updating a few images (MySQL, for example), so they can use Docker secrets. - Definitely not DDC only, but note that RBAC over secrets is a feature of the commercial product.
  - bluebeard 10y ago
    Thanks, I'm interested in trying this out.
neomantra 10y ago
This secrets infrastructure is only available at run-time and with Swarm? Are you looking to handle build-time secrets?
- diogomonicapt 10y ago
  Yes: https://github.com/docker/docker/pull/30637 https://github.com/docker/docker/pull/30637
  - neomantra 10y ago
    OMG, so much smoother than what I do now. Thanks for the link.
himanshuy 10y ago
Is Docker PCI compliant? My application deals with a lot of PII data. There was a lot of concern in my team over SOC-2 and PCI compliance when I suggested using Docker. We also talked to one of the solution architects from Amazon and he wasn't sure if Docker is ready for PII data.
- diogomonicapt 10y ago
  You can make a PCI compliant installation w/ Docker yes.
- sarnowski 10y ago
  A single software piece can't be PCI compliant but only how you use it. You have to argue that Docker uses the Linux isolation mechanisms that make those containers virtual machines in the sense and spirit of PCI. Treating containers as VMs makes some other requirements even easier like the request to have a minimal system and to only have one function per server - thats how you want containers to work anyway. (Btw PCI has nothing to do with PII.)
- Diederich 10y ago
  > solution architects from Amazon and he wasn't sure if Docker is ready for PII data. I'm curious if you can relate what his reasoning was? Docker is, among other things, a wrapper around a bunch of Linux kernel functions, the likes of which have been used for many, many years by companies like Google to facilitate all kinds of useful isolation.
  - paulddraper 10y ago
    I think it's pretty well agreed that multi-tenant Docker isn't a good security solution. Use it for for operational isolation, not for critical security isolation.
    - Diederich 10y ago
      That's what I've heard as well, and I agree with that assessment, given that the requirement is to separate untrusted code execution. At the same time, going back to the initial quote: > solution architects from Amazon and he wasn't sure if Docker is ready for PII data A lot of companies are using containers to execute code that manages all kinds of regulated data right now, ya? The security limitations of Linux namespaces and friends are mostly related to the execution of untrusted code.
mikegerwitz 10y ago
The docs explicitly state "You cannot remove a secret that a running service is using". I'm not comfortable with sensitive secrets remaining plaintext in memory after the running service possibly does not need it anymore (e.g. maybe the service only needs it on startup). It seems like you could rotate the key with something else, but that doesn't seem ideal. Or can the container itself remove the secret "file"? Consequently, because it's mounted as a filesytem, what if the service is compromised and vulnerable to arbitrary code execution, directory traversal, etc? The secret could then be leaked. Am I misinterpreting something? How would others here handle this? Edit: To clarify: rotating a secret will cause the service to restart. So I guess by "doesn't seem ideal", I mean it doesn't seem like an option.
- cpuguy83 10y ago
  You can update a service to not use a secret once it's not needed. Of course this will restart the task(s) I think. Definitely worth thinking about. Maybe a single-read secret or something. If a service is compromised, you should always assume the secret is compromised.
  - mikegerwitz 10y ago
    > Maybe a single-read secret or something. I like this idea. As a benefit, you wouldn't have to rely on developers of individual services to make sure the secret is cleared. This would also have the benefit of causing an error if something else happened to read the secret before the intended service, indicating a possible compromise (or maybe just a misconfiguration). (I use a similar concept with `xclip -l' to allow reading passwords from the clipboard a single time before it quits.)
sandGorgon 10y ago
is anyone running this stuff in production yet ? not just secrets.. but the whole Docker Swarm with the v3 compose file format thing ? We have been testing out production clusters with Swarm 1.13 and I think its neat. But what do I know - I'm not a cloud hosting specialist. And the lack of success stories out there is making us nervous.
- kossae 10y ago
  I can honestly say the only thing we found the full Docker line up (Containers, Swarm, Machine, etc.) good for large scale dev or staging/testing stuff. There are definitely uses for each of these tools, or other combinations, that have been successful for other people. It seems like Docker has a lot to focus on keeping up with the superior orchestration platforms that are out now. This in turn means quickly outdated (or missing) docs, inconsistent behavior of certain features, and other little "gotchas". Depending on the size of your team, I would recommend Kontena (www.kontena.io) or Kubernetes (https://kubernetes.io/ https://kubernetes.io/). While I haven't used the latter, a group of two devs is re-architecting our entire system to use Docker and Kontena. Nearing the end, the whole process seemed incredibly fast to iterate. While K8s seems far more proven in real production environments, it seemed much more daunting to us with more features than we needed for our use case.
  - raesene9 10y ago
    I'd recommend people read the "Risks" section of Kubernetes Docs on secrets before using it... https://kubernetes.io/docs/user-guide/secrets/#risks https://kubernetes.io/docs/user-guide/secrets/#risks. secrets stored in plain text and (by default) transmitted in plain text between etcd services, does not fill me with confidence on their security.
    - cpitman 10y ago
      Both of these are easy to fix. Configure etcd with peer certificates for clustering and a cert for client-server connections (https://coreos.com/etcd/docs/latest/v2/security.html https://coreos.com/etcd/docs/latest/v2/security.html). If you need encryption at rest, encrypt the filesystem. If setting up a secure cluster is daunting, then use a distribution that handles it for you. OpenShift (https://www.openshift.org/ https://www.openshift.org/) is built on kubernetes, and it's install is secure by default. Disclaimer: I work for Red Hat, and spend lots of time on OpenShift consulting.
m_mueller 10y ago
This is a bit off-topic, but I'll hope for the second "A" in AMA: I still have a bit trouble figuring out how docker envisions URL routing to work. Currently we use a simple setup with Hipache that forwards to our hosts running containers und multiple ports. This would work for failover and scaling as well, however at the point where DBs come into play it gets troublesome because hipache just routes in a round-robin manner instead of keeping routes persistent for each client IP (so you could end up showing them different replicas that are not in sync yet). Anyways, my issue is: Does Swarm integrate a solution for URL routing and if so, does it tackle the persistency problem?
- nicolaka 10y ago
  In the commercial product(DDC) there is something called HTTP Routing Mesh that basically adds URL routing to backend services using the underlying routing mesh in Swarm. It supports both HTTP and HTTPS ( w/SNI). It adds RBAC too to make sure only permitted services can be exposed externally. More details here: https://success.docker.com/Datacenter/Apply/Docker_Reference_Architecture%3A_Universal_Control_Plane_2.0_Service_Discovery_and_Load_Balancing#The_HTTP_Routing_Mesh https://success.docker.com/Datacenter/Apply/Docker_Reference... Hope this helps!

unpythonic 10y ago

This looks amazing. Solving secret distribution across containers will be very useful. Being able to see via `docker secret` exactly which services are using which secrets is an unexpected treat. I'm a little worried about two aspects of what has been shown. From the article it shows: $ docker exec $(docker ps --filter name=redis -q) ls -l /run/secrets total 4 -r--r--r-- 1 root root 17 Dec 13 22:48 my_secret_data From this we can tell exactly how long the secret is. If the secret service didn't do it for us, I'd like for the secrets to be null-padded to a uniform 2-4K of bytes. I'm also a bit worried that the default protection on the file has it set to world-readable. Since it appears that secret distribution is independent of the container setup itself, there doesn't appear to be any way of setting ownership and permissions on this file. That is, if one were able to chmod/chown the file in the Dockerfile, running a `docker service update --secret-rm` and `docker service update --secret-add` would reset such 'fixes'. A great start, and I can't wait to start using it.

cyli 10y ago
You can specify the UID, GID, and mode of the secret from within the container: https://docs.docker.com/engine/reference/commandline/service_create/#/create-a-service-with-secrets https://docs.docker.com/engine/reference/commandline/service..., and pass the same configuration to the service update command as well.
- unpythonic 10y ago
  Oh, that's great to know. That coupled with filling out the size to an uninteresting length with nulls (or another delimiter) covers my concerns. For example: $ echo "my secret" | dd bs=512 conv=sync | docker secret create my_secret_data -

simplehuman 10y ago

This is really nice. What is the equivalent in the kubernetes world?

eicnix 10y ago
Kubernetes supported secrets for nearly 2 years: https://kubernetes.io/docs/user-guide/secrets/ https://kubernetes.io/docs/user-guide/secrets/ They currently debate over pluggable secret stores: https://github.com/kubernetes/kubernetes/issues/10439 https://github.com/kubernetes/kubernetes/issues/10439 You can use Hashicorp Vault as a third party resource to store your secrets: https://github.com/Boostport/kubernetes-vault https://github.com/Boostport/kubernetes-vault Kubernetes also supports RBAC to control access to secrets: https://kubernetes.io/docs/admin/authorization/ https://kubernetes.io/docs/admin/authorization/
- tonyhb 10y ago
  Secrets as in passwords and keys stored at rest in plain text. So not really secrets.
  - benth 10y ago
    It's not so much the plain text part that bothers me, it's the access control. Quoting the docs at https://kubernetes.io/docs/user-guide/secrets/#security-properties https://kubernetes.io/docs/user-guide/secrets/#security-prop...: "Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet. It is a planned feature to only send secrets to nodes that actually require them, to restrict the impact of a root exploit on a single node." As your cluster grows, your risk grows.

tlrobinson 10y ago

It would be neat if you could inject secrets as environment variables. Something like: --secret="my_secret:MY_SECRET" Obviously you could have a script that does MY_SECRET=$(cat /run/secrets/my_secret) ./whatever but a lot of existing containers can be configured via environment variables.

diogomonicapt 10y ago
We explicitly chose not to support secrets as ENV variables, since they are prone to being leaked (child processes inhering parent's env; easy to leak ps -e; bug reports usually include ENV of the application; core dumps include ENV of the application, etc etc)
- tlrobinson 10y ago
  Makes sense, but how do child processes not have access to the secrets files?
  - paulddraper 10y ago
    They do. That point is less valid.
  - diogomonicapt 10y ago
    My unstated assumption was that when calling something like imagetragick you would be doing the appropriate privilege dropping using setuid and setgid. This is obviously not necessarily the case, but it's at least a lot more common than people cleaning up the ENV before creating a child process.

doublerebel 10y ago

Comparing this to Vault (with which I have been very satisfied): The article mentions a single master key for cluster encryption. Are there any plans to split this as in Shamir's Secret Sharing? How do secrets and secret authorizations renew and expire? Any plans for limited-use tokens/secrets?

heroprotagonist 10y ago

This is interesting, but in my opinion, it's not quite as universally useful as some of the secret management in other security tools because you have to explicitly manage the secret within your application by reading that in-memory filesystem. I'd much prefer passing a secret as an environment variable if that can be done securely. It's possible with some tools, but not out of the box with Docker itself. eg, with one tool, you can do: # docker run --rm -e PASSWORD={supersecret_password} someimage:latest program Then '{supersecret_password}' gets replaced in the container at runtime with the value stored in the tool or from an integration with a separate dedicated secret management tool like HashiCorp Vault, and value gets masked external to the container such as when running 'docker inspect' command. The benefit is that you don't need to modify or maintain a lot of pre-packaged applications that read environment variables instead of looking to the contents of a file on the disk, so it just works out the box. However, under ordinary circumstances, you may not want to pass secrets as environment variables in docker (or at least be careful about it). A 'docker inspect' command can show any docker user the environment variable and its value, if you don't have a tool to encrypt the contents.

eicnix 10y ago
Kubernetes supports this already. You have the choice to either mount the secret into the container file system or set it as a environment variable. https://kubernetes.io/docs/user-guide/secrets/#using-secrets-as-environment-variables https://kubernetes.io/docs/user-guide/secrets/#using-secrets...
- tonyhb 10y ago
  But Kubernetes secrets are stored - and by default transmitted amongst managers - in plain text. And Kube will share secrets with all kubelets, no questions asked. They're not "secrets".
  - eicnix 10y ago
    Yes, the kubernetes secrets resource is not a replacement for a secret store. Your specific point is being worked on: https://github.com/kubernetes/kubernetes/issues/12742 https://github.com/kubernetes/kubernetes/issues/12742 You can use Vault(https://github.com/Boostport/kubernetes-vault https://github.com/Boostport/kubernetes-vault) or extend Kubernetes easily for your specific secret as a replacement if you need extra security.

daveguy 10y ago

Not sure if anyone from docker is watching, but the images are cut off at the side on mobile.

ManoMarks 10y ago
Yes, someone from Docker is watching :-). I've alerted the blog team. Thanks for letting us know.

deleted 10y ago

[deleted]

justincormack 10y ago

Comparison of this with other secretes management frameworks here https://medium.com/on-docker/secrets-and-lie-abilities-the-state-of-modern-secret-management-2017-c82ec9136a3d#.ur0pnuirw https://medium.com/on-docker/secrets-and-lie-abilities-the-s...

tonyhb 10y ago
wrote this as a comment when that article was posted to YC: tl;dr: author looks at secret management services and reviews them KeyWhiz: Provides everything you need but with complex PKI management, meaning setup and maintenance is a pain. Secure. Vault: A+ would test again. Awesome rotation policies, on-demand secret generation via backends, master key sharing. Legit and secure, everything you need but has to be configured on top of your cluster. Docker: Super easy to use, and it's built in. 10/10 would use again. Keys encrypted at rest, keys encrypted over the wire, and shared with only nodes who need them. Secure all round Kube: Totally insecure. Plaintext at rest, plaintext over the network, shared everywhere. Basically a plaintext POC
- Leon 10y ago
  Vault is really wonderful to work with, and its integration with consul makes it phenomenally powerful.
  - jsfitzsimmons 10y ago
    What kind of stuff do you get from the integration? I mean, aside from the obvious stuff like a clustered storage backend and leader election?

lyschoening 10y ago

Will we be able to use this on Docker Cloud soon?

justincormack 10y ago
I believe it is coming soon yes.

mnm1 10y ago

Is there a way to manage secrets during the build process itself without specifying them on the command line / docker compose files?