7 ms·

Oooph, good luck. And when you have time please make Docker aware that this well known foot-gun has finally done serious harm. They have known and ignored for

Oooph, good luck. And when you have time please make Docker aware that this well known foot-gun has finally done serious harm. They have known and ignored for years that iptables/ufw on Linux is totally broken and wide open when using Docker: https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737

gnyman 5y ago
Glad someone else highlighted this old ticket. I bet this, in combination with the extremely irresponsible solution to ship mongodb without auth as default has caused countless of data leaks and destruction events. We just haven't heard about most of them. Elastic provides the same foot-gun. Last year someone deleted almost 4000 open mongodb and elastic databases in what was called the Meow attack [1]. In my opinion it's as irresponsible as HW manufacturers shipping with default passwords, something which finally got the attention of regulators[2]. So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out. I have for some time been using the data-oil analogy to describe where we are at. If data is the new oil, and a database is a tanker, then we are at the single hull tanker stage. We need double hulls, but just like the oil industry, the sw industry has little incentive to fix it themselves. I am hoping we get some regulation which improves the situation, because otherwise this will keep happening. 1: https://www.bleepingcomputer.com/news/security/new-meow-attack-has-deleted-almost-4-000-unsecured-databases/ https://www.bleepingcomputer.com/news/security/new-meow-atta... 2: https://techcrunch.com/2018/10/05/california-passes-law-that-bans-default-passwords-in-connected-devices/ https://techcrunch.com/2018/10/05/california-passes-law-that...
- qbasic_forever 5y ago
  Yeah, it seems like there's a weird inbetween phase when projects go from "awesome tool used and loved by some core people" to "this is the new normal, it's everywhere" where these issues get lost. I could see back in 2014 moby not really feeling like the quirks of ufw & iptables were its problem. But now in 2021 with how many millions of times docker run is used per day on machines all across the internet... it's just irresponsible security. It sucks, it isn't really docker's problem in the first place... but it's reality and someone needs to grab the hot potato and keep it from burning people for the good of us all.
  - jbrot 5y ago
    I know the CICD code I wrote and manage at work launches approximately a million docker containers a day, so I suspect the total number of docker containers used per day is well into the billions.
    - mafro 5y ago
      Do tell about where you would need a million docker images for a CICD pipeline.. It's either many apps, or some very complicated pipelines
      - zbentley 5y ago
        Or many builds of the same app. Think compatibility matrix testing: take one suite of tests and run them against every one of hundreds of permutations of versions of their dependencies, one container each. Not GP, just hypothesizing. This is what Travis matrixes do, for example.
      - jbrot 5y ago
        We have a suite of about two thousand integration tests and we run each test in its own docker container for isolation purposes. Multiply by 500 jobs a day and you get to a million containers.
  - gnyman 5y ago
    I agree, there is a huge difference what one should expect from a early OS project and what one should be able to expect from billion dollar companies. But it seems that without external pressure (regulatory?), it's not getting fixed at either end.
- 1337shadow 5y ago
  Ok so now it's the manufacturer's fault if you expose containers without authentication on internet, that users are fine with just "believing" that their firewall works as they expect without even testing: that's perfectly fine, manufacturer's responsibility! Except with GDPR, which makes you responsible for having regular security testing, I mean, just an nmap after changing your infra, it takes 2 minutes, and yet when you don't take these 2 minutes you find a way to blame the manufacturer. I mean, don't you /know/ that bots are scanning the internet for open unauthenticated services, and still want to be taken seriously. Back in the days every kid was doing it to host warez you know.
  - gnyman 5y ago
    I am definitely not arguing users should not be responsible for what they do. My point is that companies (especially billion dollar companies) can, and should be (more) secure by default. I am also making the observation that this does not seem to get fixed by itself, rather it seems it needs regulation, which includes GDPR.
- kortilla 5y ago
  > So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out. Wow, the level of entitlement around open source is appalling. People download and run shit for free and get mad that they didn’t configure it right and want the government to go after the maintainers. I can’t wait for the day where us open source devs have to contribute patches via pseudonyms and tor because they aren’t “government compliant”.
  - kstenerud 5y ago
    This isn't an entitlement issue; it's a public safety issue. If some philanthropist were to give out free bicycles to everyone, but it turns out that unless you tighten a bolt one of the main support bars will likely snap and could even impale you, the government would rightfully go after them regardless of any "without warranty" disclaimer or EULA. This is already well-trodden legal ground in the physical space; it's just a matter of bringing it into the virtual space.
    - tluyben2 5y ago
      It will kill open source. And a lot of other small company software development.
      - scrollaway 5y ago
        No it won't. The example GP has given didn't kill any industry either. Open source projects seldom get popular overnight. Responsibilities grow as you get more popular, that is all...
        tluyben2 5y ago
        Software is very different than making bicycles. Anyone can make an open source project in their attic and promote it. You cannot do that with most (or any) other of those other examples as it takes actual factories etc. That is the GP example. If I make 1 bike by hand and put that somewhere on the street, and someone takes it and breaks their legs, that is not my fault. Nor should it be. If I sell that 1 bike, that can be another story. It will kill open source imho as if you, in your underwear in the attic are responsible for some code you write and put on GitHub, you won't write that code anymore because of the risk; but I don't want to find out who is right here. Responsibilities should befall the person deploying the software. Aka if you write some software and throw in on GitHub, that should not do anything. But if I take that software and put it in my app or backend and takes people's money because of some bug, then I am the one who is responsible. And this is already the case anyway. No changes are needed. I think we agree though as I think we are talking about the person who writes and deploys/sells the software: they should indeed be responsible and already are.
- alias_neo 5y ago
  I spun up Mongo on a cloud VM a while back to assess viability/suitability, it was Meow'd within 30 seconds, absolutely insane. I shut it down and moved on, we don't use Mongo to date.
  - jd_mongodb 5y ago
    Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec... You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default.
    - pritambaral 5y ago
      > You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default. Serious: Listening on localhost-only works in dev environments only. In production, it is not the norm to run the application on the same host as Mongo, especially given what a resource hog Mongo is. So, for practical purposes, listen-on-localhost is actually an obstacle is needs to be disabled first-thing anyway. You guys know this too, because you do exactly this (and a lot more) on that Atlas thing you guys love to upsell everyone and their grandmother on. Honestly, it is telling that this is the only defence you could provide — that you listen on localhost — and not anything _actually_ secure in prod. Must come in handy when upselling Atlas, I guess, that your the default configuration conveniently omits everything. > Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-chec https://docs.mongodb.com/manual/administration/security-chec... Snark: maybe they couldn't trust your guidelines because MongoDB the company is a known blatant liar[1]. 1: https://twitter.com/jepsen_io/status/1255867265997844484 https://twitter.com/jepsen_io/status/1255867265997844484
      - jd_mongodb 5y ago
        There wasn't a lot of information in your previous post. As I pointed out there are a comprehensive set of guidelines for enforcing security. Our defaults make it difficult to accidentally expose your data these days. However if you do add a MongoDB database to a public IP address we strongly encourage you to add a strong password. Better still do not expose your database on the public internet. Put it behind a firewall with auth enabled, secure it with a certificate and only allow access to named IP addresses.
- Aeolun 5y ago
  I’d argue that anyone that uses Mongo given those defaults is equally culpable
  - anotherhue 5y ago
    Mongo recruiters often reach out to me, I tell them that I'd never get that stain off my resume.
sneak 5y ago
This massive PITA is why I switched to using Hetzner's upstream colo firewall instead of on-host iptables on my boxes there running Docker. It's lame and also it's not even stateful, but at least the garbage software coming out of Docker Inc these days can't accidentally open my system up to the world. The userspace proxy used by Docker Swarm is still trashing my L3 client IPs though, yet another docker networking hassle. Ultimately solely relying on an L3 firewall for access control is a bad idea, though. Your services should all have authentication turned on, even if they are only bound to localhost, for reasons which at this moment must be obvious.
alias_neo 5y ago
We simply stopped using Docker's control of the firewall. It also takes docker-proxy out of the equation so you have far more control and it's less resource intensive (docker starts a proxy instance per port exposed). I've commented here in the past about my feelings on Docker's attitude towards real issues raised by users, the only way I can describe it succinctly is "contempt". There's some serious flaws, with tickets that are the best part a decade old now, one simply can't rely on them to do anything. Stop using Docker or solve the issue yourself is my advice.
adevx 5y ago
This has also hit me hard when it opened up Mailcow's not open relay protected internal network. Creating an open relay, and almost getting me banned by Hetzner. I try to avoid Docker when not absolutely necessary just to keep my stack as simple as possible.
patrakov 5y ago
It is 100% unfair to blame Docker for this foot-gun, especially because they cannot really do anything to fix it, because that's how the firewall works in the Linux kernel. Look: Podman has exactly the same issue when not running rootless. The root cause is that IP packets going to the containers are not going through the INPUT chain of the "filter" table (they go through FORWARD), while various firewall projects like ufw or firewalld only provide convenient management of the INPUT chain, and, worse, don't event provide a good way to express the notion of "packets going to external port 27017 and then forwarded" (i.e. the equivalent of the --ctorigdst option provided by raw iptables). Here are some options for container engines: a) Use only a userspace proxy (which is what Docker does when configured with {"iptables": false}). This way, there is no packet forwarding, so packets go through the INPUT chain, just as expected by high-level firewall packages like ufw or firewalld. The major downside (which makes this method useless e.g. for containerized mail servers) is that the information about the source IP is lost, and there is no good way to fix this. I guess TPROXY can help here, but nobody uses it. b) Use slirp4netns (which is what Podman does when running rootless). It has a really nice mode (available via "podman run --network=slirp4netns:port_handler=slirp4netns") where on the host side, there is only a userspace process listening (so that packets go through the INPUT chain), but inside the container, packets going out of tap0 have the correct source IP. The downside (actually a Podman limitation) is that you can't set up multiple containers communicating over internal IPs. I would say that I am not really in favor of options (a) and (b) because of the overhead created by the proxy or by slirp4netns. If port forwarding can be done in the kernel (and it can, the only missing piece is --ctorigdst in high-level firewalls), it should be done in the kernel. c) Document the situation better (e.g. I don't see the --ctorigdst option mentioned at all on https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/), shift the blame to firewall authors so that they start creating a duplicate of each INPUT rule also in the FORWARD chain with --ctorigdst added as necessary. d) Provide usable primitives (similar to Network Policies in Kubernetes) to control the firewall for containers, so that ufw or firewalld is not needed.
- geenat 5y ago
  Horribly contrived- but you can run rootless docker with slirp4netns if you: * Install slirp4netns * DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER="slirp4netns" * Upgrade to rootless docker 20.10.7 or higher (was bugged in 20.10.6. Yup.. bad.)

gnyman 5y ago

Glad someone else highlighted this old ticket. I bet this, in combination with the extremely irresponsible solution to ship mongodb without auth as default has caused countless of data leaks and destruction events. We just haven't heard about most of them. Elastic provides the same foot-gun. Last year someone deleted almost 4000 open mongodb and elastic databases in what was called the Meow attack [1]. In my opinion it's as irresponsible as HW manufacturers shipping with default passwords, something which finally got the attention of regulators[2]. So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out. I have for some time been using the data-oil analogy to describe where we are at. If data is the new oil, and a database is a tanker, then we are at the single hull tanker stage. We need double hulls, but just like the oil industry, the sw industry has little incentive to fix it themselves. I am hoping we get some regulation which improves the situation, because otherwise this will keep happening. 1: https://www.bleepingcomputer.com/news/security/new-meow-attack-has-deleted-almost-4-000-unsecured-databases/ https://www.bleepingcomputer.com/news/security/new-meow-atta... 2: https://techcrunch.com/2018/10/05/california-passes-law-that-bans-default-passwords-in-connected-devices/ https://techcrunch.com/2018/10/05/california-passes-law-that...

qbasic_forever 5y ago
Yeah, it seems like there's a weird inbetween phase when projects go from "awesome tool used and loved by some core people" to "this is the new normal, it's everywhere" where these issues get lost. I could see back in 2014 moby not really feeling like the quirks of ufw & iptables were its problem. But now in 2021 with how many millions of times docker run is used per day on machines all across the internet... it's just irresponsible security. It sucks, it isn't really docker's problem in the first place... but it's reality and someone needs to grab the hot potato and keep it from burning people for the good of us all.
- jbrot 5y ago
  I know the CICD code I wrote and manage at work launches approximately a million docker containers a day, so I suspect the total number of docker containers used per day is well into the billions.
  - mafro 5y ago
    Do tell about where you would need a million docker images for a CICD pipeline.. It's either many apps, or some very complicated pipelines
    - zbentley 5y ago
      Or many builds of the same app. Think compatibility matrix testing: take one suite of tests and run them against every one of hundreds of permutations of versions of their dependencies, one container each. Not GP, just hypothesizing. This is what Travis matrixes do, for example.
    - jbrot 5y ago
      We have a suite of about two thousand integration tests and we run each test in its own docker container for isolation purposes. Multiply by 500 jobs a day and you get to a million containers.
- gnyman 5y ago
  I agree, there is a huge difference what one should expect from a early OS project and what one should be able to expect from billion dollar companies. But it seems that without external pressure (regulatory?), it's not getting fixed at either end.
1337shadow 5y ago
Ok so now it's the manufacturer's fault if you expose containers without authentication on internet, that users are fine with just "believing" that their firewall works as they expect without even testing: that's perfectly fine, manufacturer's responsibility! Except with GDPR, which makes you responsible for having regular security testing, I mean, just an nmap after changing your infra, it takes 2 minutes, and yet when you don't take these 2 minutes you find a way to blame the manufacturer. I mean, don't you /know/ that bots are scanning the internet for open unauthenticated services, and still want to be taken seriously. Back in the days every kid was doing it to host warez you know.
- gnyman 5y ago
  I am definitely not arguing users should not be responsible for what they do. My point is that companies (especially billion dollar companies) can, and should be (more) secure by default. I am also making the observation that this does not seem to get fixed by itself, rather it seems it needs regulation, which includes GDPR.
kortilla 5y ago
> So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out. Wow, the level of entitlement around open source is appalling. People download and run shit for free and get mad that they didn’t configure it right and want the government to go after the maintainers. I can’t wait for the day where us open source devs have to contribute patches via pseudonyms and tor because they aren’t “government compliant”.
- kstenerud 5y ago
  This isn't an entitlement issue; it's a public safety issue. If some philanthropist were to give out free bicycles to everyone, but it turns out that unless you tighten a bolt one of the main support bars will likely snap and could even impale you, the government would rightfully go after them regardless of any "without warranty" disclaimer or EULA. This is already well-trodden legal ground in the physical space; it's just a matter of bringing it into the virtual space.
  - tluyben2 5y ago
    It will kill open source. And a lot of other small company software development.
    - scrollaway 5y ago
      No it won't. The example GP has given didn't kill any industry either. Open source projects seldom get popular overnight. Responsibilities grow as you get more popular, that is all...
      - tluyben2 5y ago
        Software is very different than making bicycles. Anyone can make an open source project in their attic and promote it. You cannot do that with most (or any) other of those other examples as it takes actual factories etc. That is the GP example. If I make 1 bike by hand and put that somewhere on the street, and someone takes it and breaks their legs, that is not my fault. Nor should it be. If I sell that 1 bike, that can be another story. It will kill open source imho as if you, in your underwear in the attic are responsible for some code you write and put on GitHub, you won't write that code anymore because of the risk; but I don't want to find out who is right here. Responsibilities should befall the person deploying the software. Aka if you write some software and throw in on GitHub, that should not do anything. But if I take that software and put it in my app or backend and takes people's money because of some bug, then I am the one who is responsible. And this is already the case anyway. No changes are needed. I think we agree though as I think we are talking about the person who writes and deploys/sells the software: they should indeed be responsible and already are.
alias_neo 5y ago
I spun up Mongo on a cloud VM a while back to assess viability/suitability, it was Meow'd within 30 seconds, absolutely insane. I shut it down and moved on, we don't use Mongo to date.
- jd_mongodb 5y ago
  Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec... You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default.
  - pritambaral 5y ago
    > You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default. Serious: Listening on localhost-only works in dev environments only. In production, it is not the norm to run the application on the same host as Mongo, especially given what a resource hog Mongo is. So, for practical purposes, listen-on-localhost is actually an obstacle is needs to be disabled first-thing anyway. You guys know this too, because you do exactly this (and a lot more) on that Atlas thing you guys love to upsell everyone and their grandmother on. Honestly, it is telling that this is the only defence you could provide — that you listen on localhost — and not anything _actually_ secure in prod. Must come in handy when upselling Atlas, I guess, that your the default configuration conveniently omits everything. > Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-chec https://docs.mongodb.com/manual/administration/security-chec... Snark: maybe they couldn't trust your guidelines because MongoDB the company is a known blatant liar[1]. 1: https://twitter.com/jepsen_io/status/1255867265997844484 https://twitter.com/jepsen_io/status/1255867265997844484
    - jd_mongodb 5y ago
      There wasn't a lot of information in your previous post. As I pointed out there are a comprehensive set of guidelines for enforcing security. Our defaults make it difficult to accidentally expose your data these days. However if you do add a MongoDB database to a public IP address we strongly encourage you to add a strong password. Better still do not expose your database on the public internet. Put it behind a firewall with auth enabled, secure it with a certificate and only allow access to named IP addresses.
Aeolun 5y ago
I’d argue that anyone that uses Mongo given those defaults is equally culpable
- anotherhue 5y ago
  Mongo recruiters often reach out to me, I tell them that I'd never get that stain off my resume.

sneak 5y ago

This massive PITA is why I switched to using Hetzner's upstream colo firewall instead of on-host iptables on my boxes there running Docker. It's lame and also it's not even stateful, but at least the garbage software coming out of Docker Inc these days can't accidentally open my system up to the world. The userspace proxy used by Docker Swarm is still trashing my L3 client IPs though, yet another docker networking hassle. Ultimately solely relying on an L3 firewall for access control is a bad idea, though. Your services should all have authentication turned on, even if they are only bound to localhost, for reasons which at this moment must be obvious.

alias_neo 5y ago

We simply stopped using Docker's control of the firewall. It also takes docker-proxy out of the equation so you have far more control and it's less resource intensive (docker starts a proxy instance per port exposed). I've commented here in the past about my feelings on Docker's attitude towards real issues raised by users, the only way I can describe it succinctly is "contempt". There's some serious flaws, with tickets that are the best part a decade old now, one simply can't rely on them to do anything. Stop using Docker or solve the issue yourself is my advice.

adevx 5y ago

This has also hit me hard when it opened up Mailcow's not open relay protected internal network. Creating an open relay, and almost getting me banned by Hetzner. I try to avoid Docker when not absolutely necessary just to keep my stack as simple as possible.

patrakov 5y ago

It is 100% unfair to blame Docker for this foot-gun, especially because they cannot really do anything to fix it, because that's how the firewall works in the Linux kernel. Look: Podman has exactly the same issue when not running rootless. The root cause is that IP packets going to the containers are not going through the INPUT chain of the "filter" table (they go through FORWARD), while various firewall projects like ufw or firewalld only provide convenient management of the INPUT chain, and, worse, don't event provide a good way to express the notion of "packets going to external port 27017 and then forwarded" (i.e. the equivalent of the --ctorigdst option provided by raw iptables). Here are some options for container engines: a) Use only a userspace proxy (which is what Docker does when configured with {"iptables": false}). This way, there is no packet forwarding, so packets go through the INPUT chain, just as expected by high-level firewall packages like ufw or firewalld. The major downside (which makes this method useless e.g. for containerized mail servers) is that the information about the source IP is lost, and there is no good way to fix this. I guess TPROXY can help here, but nobody uses it. b) Use slirp4netns (which is what Podman does when running rootless). It has a really nice mode (available via "podman run --network=slirp4netns:port_handler=slirp4netns") where on the host side, there is only a userspace process listening (so that packets go through the INPUT chain), but inside the container, packets going out of tap0 have the correct source IP. The downside (actually a Podman limitation) is that you can't set up multiple containers communicating over internal IPs. I would say that I am not really in favor of options (a) and (b) because of the overhead created by the proxy or by slirp4netns. If port forwarding can be done in the kernel (and it can, the only missing piece is --ctorigdst in high-level firewalls), it should be done in the kernel. c) Document the situation better (e.g. I don't see the --ctorigdst option mentioned at all on https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/), shift the blame to firewall authors so that they start creating a duplicate of each INPUT rule also in the FORWARD chain with --ctorigdst added as necessary. d) Provide usable primitives (similar to Network Policies in Kubernetes) to control the firewall for containers, so that ufw or firewalld is not needed.

geenat 5y ago
Horribly contrived- but you can run rootless docker with slirp4netns if you: * Install slirp4netns * DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER="slirp4netns" * Upgrade to rootless docker 20.10.7 or higher (was bugged in 20.10.6. Yup.. bad.)