5 ms·

This is why you shouldn't be using Docker in production. It's a great tool, but it's simply not designed for that kind of environment. Edit: note I said Docker

This is why you shouldn't be using Docker in production. It's a great tool, but it's simply not designed for that kind of environment. Edit: note I said Docker specifically, nothing about containerization.

wkdneidbwf 4y ago
what. docker isn’t the problem here. dbs on public subnets, and the lack of monitoring for accidental db exposure are the actual issues here.
- Kwpolska 4y ago
  And running a database without authentication.
  - chrismeller 4y ago
    True, but even the author covered that. No one has mentioned Docker.
- yjftsjthsd-h 4y ago
  > docker isn’t the problem here I mean... if they weren't using docker it would have been fine, but because they used docker it wasn't fine. That reads like docker is the problem. That further layers could have mitigated it doesn't make docker not the problem.
  - luckylion 4y ago
    If they had understood how docker works, it would've been fine, too. But because they didn't and used a software firewall as their only line of defense and didn't bother with authentication for the DB server, it wasn't fine.
    - yjftsjthsd-h 4y ago
      > If they had understood how docker works, it would've been fine, too. They shouldn't have needed to, for this. > But because they didn't and used a software firewall as their only line of defense Okay? That should have been safe. Like, sure, there are ways to add more layers, but that layer shouldn't have failed them. > didn't bother with authentication for the DB server, it wasn't fine. They shouldn't have needed to. Like yes, running a DB without authentication isn't a great idea, but they had good reason to believe that it wouldn't be exposed.
      - luckylion 4y ago
        Running a server is a lot like doing electrical work. If you don't really know what you're doing and rely on vague intuition about how things should probably work, you might be in for a shock.
  - DistractionRect 4y ago
    Where does the buck stop? This was simply a few layers of bad configuration. They didn't secure their database with auth/access control and they misconfigured docker. They have a few things at their disposal: - Using the docker-user chain to set firewall rules - Running docker such that the default bind address for the port directive is 127.0.0.1 instead of 0.0.0.0. This puts a safety on the footgun. - Explicitly setting the bind address of the port directive when bringing up the container. Docker didn't come along, install itself, and open up the port. The sysadmin did. It's unfortunate they didn't know how it interacted with the firewall or how to properly configure it, but given that, should they really have been rolling it out to production systems? They tested on prod and learned in trial by fire.
    - yjftsjthsd-h 4y ago
      > They didn't secure their database with auth/access control True, although they had reason to believe that that was safe. > they misconfigured docker. Ah, no, that's where we disagree. They didn't configure it, docker shipped an insane default that bypasses existing security measures. There is absolutely no reason to expect that running a program in docker magically makes it ignore the system firewall.
      - DistractionRect 4y ago
        The sysadmin is responsible for what they install and how it's configured. Docker didn't specify what ports to open, what container to run, volume mounts, image etc. That's part of the config the sysadmin supplies when they spin it up. No matter how you slice it, it down to them not understanding what they were pushing to prod. By the same token having an unsecured database is a bad default, so we can keep passing the buck around.
      - xboxnolifes 4y ago
        > True, although they had reason to believe that that was safe. That's like the antithesis of layered security. There would be no point in layers if you assume a single layer will protect you.
  - wkdneidbwf 4y ago
    it wouldn’t have been fine—they didn’t realize the configuration mistake until their database had been dropped. there are plenty of scenarios that could have led to the exact same outcome. i’d argue it would have been a matter of time. the problem is that it was possible to accidentally expose their credential-less db to the internet _at all_ and that they had no monitoring or tools in place to detect the misconfiguration. that’s a design flaw, and again, not a docker-specific problem.
  - remram 4y ago
    By that logic you could say that iptables/netfilter was the problem. Or Linux. Or maybe the IP protocol.
    - yjftsjthsd-h 4y ago
      I don't think so; at worst, each of those is passive and might fail to make you more secure. Docker goes out of its way to add holes to existing security. It's like... if iptables decided to ship a feature that detected when it was running in AWS and "helpfully" automatically reconfigured your security groups to allow any traffic that was allowed in iptables, that would be the same.
      - remram 4y ago
        I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host. Docker could not extend the host's firewall to containers without changing how iptables work.
        yjftsjthsd-h 4y ago
        > I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). That's exactly what happens. Docker sticking its rules before the normal filters is exactly what I mean when I say it bypasses the firewall rules. Like... you're literally describing the implementation details of what I said it does. > The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host. They clearly are? If a docker container was listening but not on a port on the host's internet-facing interface (indirected though it may be), none of this would be a problem. The problem is precisely that if you have a host firewall rule that says "this port is blocked", docker will "helpfully" preempt that and connect that port to a container, which is a massive and unreasonable footgun. > Docker could not extend the host's firewall to containers without changing how iptables work. podman seems to manage fine, so I have trouble believing that.
bornfreddy 4y ago
I respectfully disagree. It solves a huge problem - reproducible app environment. It comes with its share of footguns too (hint: always explicitly specify pool of IP addresses it can use!), yes, but what doesn't?
- convolvatron 4y ago
  its probably not impossible to do this without the footguns
  - bornfreddy 4y ago
    True, it just wasn't done in this particular universe we live in. :shrugs: Still worth using it in production, but being careful.
- chrismeller 4y ago
  I didn’t say he shouldn’t use containerization. He just shouldn’t have used Docker. Docker has always been very dev environment focused.
  - bornfreddy 4y ago
    Are you aware of another containerization technology that would be more suitable? (genuinly curious - I only have experience with Docker and never felt the need to look elsewhere, even with footguns, but I'm still curious)
    - KronisLV 4y ago
      Some will suggest that Podman is a good choice: https://podman.io/ https://podman.io/ Though personally I would miss Docker Swarm which comes with Docker by default and, in my opinion, is a nice bridge between something like Docker Compose (easy single node deployments) and Nomad/Kubernetes (both more complex in comparison, need separate setup and lots of admin work). Personally I also think that Docker is sufficient in most cases and is pretty boring and stable. Docker Desktop might irk some, but seems like they wanted that revenue. However, for desktop GUI, Podman has Podman Desktop: https://podman-desktop.io/ https://podman-desktop.io/ Or one can also consider Rancher Desktop as another GUI tool, if desired: https://rancherdesktop.io/ https://rancherdesktop.io/

wkdneidbwf 4y ago

what. docker isn’t the problem here. dbs on public subnets, and the lack of monitoring for accidental db exposure are the actual issues here.

Kwpolska 4y ago
And running a database without authentication.
- chrismeller 4y ago
  True, but even the author covered that. No one has mentioned Docker.
yjftsjthsd-h 4y ago
> docker isn’t the problem here I mean... if they weren't using docker it would have been fine, but because they used docker it wasn't fine. That reads like docker is the problem. That further layers could have mitigated it doesn't make docker not the problem.
- luckylion 4y ago
  If they had understood how docker works, it would've been fine, too. But because they didn't and used a software firewall as their only line of defense and didn't bother with authentication for the DB server, it wasn't fine.
  - yjftsjthsd-h 4y ago
    > If they had understood how docker works, it would've been fine, too. They shouldn't have needed to, for this. > But because they didn't and used a software firewall as their only line of defense Okay? That should have been safe. Like, sure, there are ways to add more layers, but that layer shouldn't have failed them. > didn't bother with authentication for the DB server, it wasn't fine. They shouldn't have needed to. Like yes, running a DB without authentication isn't a great idea, but they had good reason to believe that it wouldn't be exposed.
    - luckylion 4y ago
      Running a server is a lot like doing electrical work. If you don't really know what you're doing and rely on vague intuition about how things should probably work, you might be in for a shock.
- DistractionRect 4y ago
  Where does the buck stop? This was simply a few layers of bad configuration. They didn't secure their database with auth/access control and they misconfigured docker. They have a few things at their disposal: - Using the docker-user chain to set firewall rules - Running docker such that the default bind address for the port directive is 127.0.0.1 instead of 0.0.0.0. This puts a safety on the footgun. - Explicitly setting the bind address of the port directive when bringing up the container. Docker didn't come along, install itself, and open up the port. The sysadmin did. It's unfortunate they didn't know how it interacted with the firewall or how to properly configure it, but given that, should they really have been rolling it out to production systems? They tested on prod and learned in trial by fire.
  - yjftsjthsd-h 4y ago
    > They didn't secure their database with auth/access control True, although they had reason to believe that that was safe. > they misconfigured docker. Ah, no, that's where we disagree. They didn't configure it, docker shipped an insane default that bypasses existing security measures. There is absolutely no reason to expect that running a program in docker magically makes it ignore the system firewall.
    - DistractionRect 4y ago
      The sysadmin is responsible for what they install and how it's configured. Docker didn't specify what ports to open, what container to run, volume mounts, image etc. That's part of the config the sysadmin supplies when they spin it up. No matter how you slice it, it down to them not understanding what they were pushing to prod. By the same token having an unsecured database is a bad default, so we can keep passing the buck around.
    - xboxnolifes 4y ago
      > True, although they had reason to believe that that was safe. That's like the antithesis of layered security. There would be no point in layers if you assume a single layer will protect you.
- wkdneidbwf 4y ago
  it wouldn’t have been fine—they didn’t realize the configuration mistake until their database had been dropped. there are plenty of scenarios that could have led to the exact same outcome. i’d argue it would have been a matter of time. the problem is that it was possible to accidentally expose their credential-less db to the internet _at all_ and that they had no monitoring or tools in place to detect the misconfiguration. that’s a design flaw, and again, not a docker-specific problem.
- remram 4y ago
  By that logic you could say that iptables/netfilter was the problem. Or Linux. Or maybe the IP protocol.
  - yjftsjthsd-h 4y ago
    I don't think so; at worst, each of those is passive and might fail to make you more secure. Docker goes out of its way to add holes to existing security. It's like... if iptables decided to ship a feature that detected when it was running in AWS and "helpfully" automatically reconfigured your security groups to allow any traffic that was allowed in iptables, that would be the same.
    - remram 4y ago
      I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host. Docker could not extend the host's firewall to containers without changing how iptables work.
      - yjftsjthsd-h 4y ago
        > I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). That's exactly what happens. Docker sticking its rules before the normal filters is exactly what I mean when I say it bypasses the firewall rules. Like... you're literally describing the implementation details of what I said it does. > The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host. They clearly are? If a docker container was listening but not on a port on the host's internet-facing interface (indirected though it may be), none of this would be a problem. The problem is precisely that if you have a host firewall rule that says "this port is blocked", docker will "helpfully" preempt that and connect that port to a container, which is a massive and unreasonable footgun. > Docker could not extend the host's firewall to containers without changing how iptables work. podman seems to manage fine, so I have trouble believing that.

bornfreddy 4y ago

I respectfully disagree. It solves a huge problem - reproducible app environment. It comes with its share of footguns too (hint: always explicitly specify pool of IP addresses it can use!), yes, but what doesn't?

convolvatron 4y ago
its probably not impossible to do this without the footguns
- bornfreddy 4y ago
  True, it just wasn't done in this particular universe we live in. :shrugs: Still worth using it in production, but being careful.
chrismeller 4y ago
I didn’t say he shouldn’t use containerization. He just shouldn’t have used Docker. Docker has always been very dev environment focused.
- bornfreddy 4y ago
  Are you aware of another containerization technology that would be more suitable? (genuinly curious - I only have experience with Docker and never felt the need to look elsewhere, even with footguns, but I'm still curious)
  - KronisLV 4y ago
    Some will suggest that Podman is a good choice: https://podman.io/ https://podman.io/ Though personally I would miss Docker Swarm which comes with Docker by default and, in my opinion, is a nice bridge between something like Docker Compose (easy single node deployments) and Nomad/Kubernetes (both more complex in comparison, need separate setup and lots of admin work). Personally I also think that Docker is sufficient in most cases and is pretty boring and stable. Docker Desktop might irk some, but seems like they wanted that revenue. However, for desktop GUI, Podman has Podman Desktop: https://podman-desktop.io/ https://podman-desktop.io/ Or one can also consider Rancher Desktop as another GUI tool, if desired: https://rancherdesktop.io/ https://rancherdesktop.io/