5 ms·

We are doing something very similar (mesos rather than k8s), also in a financial organisation. Our main issue is also security - as docker containers aren't ve

We are doing something very similar (mesos rather than k8s), also in a financial organisation. Our main issue is also security - as docker containers aren't very good at actually containing, from a security PoV, we're using SELinux MLS, assigning each container a unique SELinux level/category (fedora-atomic sets this up nicely for you if anyone wants to try). This is great for security, but breaks so many things - lots of the cutting edge work with docker and related tech isn't done by people that care deeply about security, or really understand this sort of usecase, so we run into a huge amount of edgecases and bugs. We spend a lot of time fighting this. It also makes having a really good build/deploy pipeline essential, as well as a good environment set up to reproduce and diagnose issues.

je42 11y ago
Do you have links pointing at what docker containers leak from a security PoV ?
- riskable 11y ago
  This largely depends on how you're implementing them. If you're using Kubernetes, for example, all Docker containers being managed by Kubernetes can talk to each other without restrictions. From each container's perspective they're part of a big, flat network with no firewalls between them. So if you have a container running the database for client A that means that when client B spins up their little malicious container they can directly connect to client A's DB. That's one way that Docker containers can "leak" (sort of). Another way is that the filesystem for running containers is directly accessible from outside the container. That means that anyone with the power to start/stop containers has the power to read the filesystems for all other containers. There used to be a problem with, "root inside the container is the same as root outside the container" but that was fixed with Docker 1.9 via the new UID mapping feature. So the filesystem and processes running as UID 0 inside the container will actually be some high UID in the parent OS. So unless you're running Red Hat's version of Docker (which is currently hard set to be Docker 1.8!) you can upgrade and solve that problem pretty quickly with a few tweaks.
tracker1 11y ago
Along a similar grain, I'm curious if anyone's used Joyent's infrastructure stack internally... I know they offer it, but not familiar enough with it. It seems Solaris containers as a base for docker containers is a better security model, but not sure what parts, if enough is open to implement without paying consultation from Joyent to get started even. It's definitely a compelling model. I am only slightly surprised that there's still a lot of bugginess in this aspect, though I haven't had to deal with that level of security need on the infrastructure side in a number of years. I'd rather be doing app-dev over ops-dev.
- seanp2k2 11y ago
  HashiCorp also has a pretty complete solution called Atlas : https://www.hashicorp.com/ https://www.hashicorp.com/ ( different from the Netflix Atlas http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html http://techblog.netflix.com/2014/12/introducing-atlas-netfli... ) Joyent's Triton looks cool: https://www.joyent.com/ https://www.joyent.com/ as does CoreOS's Tectonic (built on top of k8s): https://tectonic.com/ https://tectonic.com/
  - tracker1 11y ago
    I've been following CoreOS with great interest, I was just thinking from a security standpoint Joyent's Triton looks amazing... I don't think their pricing is competitive with Google/AWS/Azure though. Hadn't looked at Atlas... I changed jobs into a larger company, and don't really have to deal with the ops side ever. Though eagerly awaiting a blessed docker based solution to become available in the org... it'll mean that projects can be developed more readily without being limited to the Java based infrastructure in place now.

je42 11y ago

Do you have links pointing at what docker containers leak from a security PoV ?

riskable 11y ago
This largely depends on how you're implementing them. If you're using Kubernetes, for example, all Docker containers being managed by Kubernetes can talk to each other without restrictions. From each container's perspective they're part of a big, flat network with no firewalls between them. So if you have a container running the database for client A that means that when client B spins up their little malicious container they can directly connect to client A's DB. That's one way that Docker containers can "leak" (sort of). Another way is that the filesystem for running containers is directly accessible from outside the container. That means that anyone with the power to start/stop containers has the power to read the filesystems for all other containers. There used to be a problem with, "root inside the container is the same as root outside the container" but that was fixed with Docker 1.9 via the new UID mapping feature. So the filesystem and processes running as UID 0 inside the container will actually be some high UID in the parent OS. So unless you're running Red Hat's version of Docker (which is currently hard set to be Docker 1.8!) you can upgrade and solve that problem pretty quickly with a few tweaks.

tracker1 11y ago

Along a similar grain, I'm curious if anyone's used Joyent's infrastructure stack internally... I know they offer it, but not familiar enough with it. It seems Solaris containers as a base for docker containers is a better security model, but not sure what parts, if enough is open to implement without paying consultation from Joyent to get started even. It's definitely a compelling model. I am only slightly surprised that there's still a lot of bugginess in this aspect, though I haven't had to deal with that level of security need on the infrastructure side in a number of years. I'd rather be doing app-dev over ops-dev.

seanp2k2 11y ago
HashiCorp also has a pretty complete solution called Atlas : https://www.hashicorp.com/ https://www.hashicorp.com/ ( different from the Netflix Atlas http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html http://techblog.netflix.com/2014/12/introducing-atlas-netfli... ) Joyent's Triton looks cool: https://www.joyent.com/ https://www.joyent.com/ as does CoreOS's Tectonic (built on top of k8s): https://tectonic.com/ https://tectonic.com/
- tracker1 11y ago
  I've been following CoreOS with great interest, I was just thinking from a security standpoint Joyent's Triton looks amazing... I don't think their pricing is competitive with Google/AWS/Azure though. Hadn't looked at Atlas... I changed jobs into a larger company, and don't really have to deal with the ops side ever. Though eagerly awaiting a blessed docker based solution to become available in the org... it'll mean that projects can be developed more readily without being limited to the Java based infrastructure in place now.