11 ms·

It's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying p

It's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying platform not being bonkers (but alas, it was: https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737 and https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054). Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Should the attacker have not done the attack? Without a doubt. But should docker have not completely undone a security control? Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion. Previously: https://news.ycombinator.com/item?id=27613217 https://news.ycombinator.com/item?id=27613217

Firerouge 5y ago
This same one got me after years of using Docker, I only discovered it after using the combination of Ubuntu Server (and it's ufw) on a DMZed device. I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords. I've been using https://github.com/shinebayar-g/ufw-docker-automated https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers. It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.
- cbo100 5y ago
  Yep. This exact interaction between UFW and Docker had me stumped for a while too. Totally unexpected outcome. Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.
1337shadow 5y ago
> Published ports > This creates a firewall rule which maps a container port to a port on the Docker host to the outside world. Source : https://docs.docker.com/config/containers/container-networking/ https://docs.docker.com/config/containers/container-networki... Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost". However testing your firewall after publishing a docker port seems common sense. Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/ And that's how I made a simple firewall that works: https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/main.yml#L77-115 https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/... Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it. So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017. But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".
- 411111111111111 5y ago
  > Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. That only works with local communication unless you use docker swarm. So byebye high availability. If you're going to make suggestions, at least think about them from a production point of view.
  - 1337shadow 5y ago
    Most productions are fine with 99.9% uptime, and a single server is fine for that, so HA is not necessary for most production out there. For example, I'm running a governmental, nationwide service handling 300k req/day, with 1k admins working daily on the site, on a single server without anyone complaining, and without less than 99.9% uptime even though I reboot to upgrade the kernel twice a month, I did have to fine tune the system but that was actually pretty easy.
    - yoz-y 5y ago
      That’s 10qps assuming working hours indeed you don’t need more than one server for that.
      - 1337shadow 5y ago
        How many user facing projects actually got so far they needed more than one server? 1% of them?
        bildung 5y ago
        I'd say even less - a single db server is almost enough for let's encrypt: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...
    - hyperman1 5y ago
nuker 5y ago
> They clearly made the right moves The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT. Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"
- diamondo25 5y ago
  Even Google Cloud SQL just is a managed instance with a public IP (that you don't seem to be able to make private). Firewalls always help, and thats what Cloud SQL does: whitelist your ip when using the CLI.
  - nuker 5y ago
    What I do in AWS is having a VPC with 3 subnet types, public, private and restricted. Public ones have default route to inet, private to NAT, restricted - none. DBs go in restricted, load balancers in public and web/app servers in private subnets. Plus whitelist firewalling using Security Groups.
    - mercora 5y ago
      when i evaluated using a NAT setup using AWS NAT gateways it appeared to be quite expensive. we settled for strict VPC rules using only SecurityGroups in order to restrict communication. the instances did have public ip but nothing but intended services could connect to it anyways... i.e an instance running a HTTP API could only be connected to if initiated from the load balancer and itself could only connect to relevant services such as a RDS cluster...
      - nuker 5y ago
        Normally machines behind NAT use it only for pulling stuff on boot time. Main traffic goes via LBs. depends on your case, of course.
        5y ago
mschuster91 5y ago
> Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion. It's clearly written in the docs: To expose a container’s internal port, an operator can start the container with the -P or -p flag. The exposed port is accessible on the host and the ports are available to any client that can reach the host. (from https://docs.docker.com/engine/reference/run/#expose-incoming-ports https://docs.docker.com/engine/reference/run/#expose-incomin...) It should be common knowledge by now to either create a virtual interface / network on the host to publish ports on for usage by services that aren't running in Docker or (if you are in an environment where all services are in Docker) use --link between containers.
- arp242 5y ago
  I don't think that's clear at all. If I set "bind_ip = *" in some application then it's also "available to any client that can reach the host", but the firewall is in front of that. I certainly wouldn't expect an application to frob with my firewall. And as I understand it, this is very much an unintentional side-effect of ufw and Docker interacting – it's not Docker's the intention at all to override any iptable rules, just an unfortunate side-effect. "It should be common knowledge by now" is very hand-wavy. I never used Docker much, and I could have been bitten by this. Where do I get this "common knowledge" from? Not everyone is a full-time sysadmin; some people just want to run a small service, read the basic documentation on various things etc., and set things up. Not everyone is super-invested in Docker (or any other tool for that matter) to have all the "common knowledge" that all the experts have. This is why defaults matter.
  - secabeen 5y ago
    Same here. It's even a little worse because of the vagaries of Docker networking. I have a server, it has a firewall that has all ports blocked. If I install Docker to run some containers and use the --network=host option, it all works fine. Containers listen on ports on the host, my existing firewall controls it, great. If I use what would seem like the more secure port-forward option, Docker adds a new rule to the top of my firewall ruleset opening the forwarded ports to all traffic, overriding my existing firewall for that port entirely. I get that Docker is often run by devs on their laptops, but this is bad behavior. The simplest solution I've seen so-far is Docker should only automatically set these firewall rules on devices that only have RFC1918 IPs. That would still keep things simple for devs on laptops, but would prevent firewall override on servers.
bildung 5y ago
> It's not all that great blaming the victim. Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management https://www.sebokwiki.org/wiki/Risk_Management
- eganist 5y ago
  > Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. There can be multiple victims, multiple causes/threat actors, and overlap between two categories. But what's the end user a victim of in this case? Other than a brief lack of availability?
  - bildung 5y ago
    Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring? Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
    - eganist 5y ago
      > Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring? I don't understand the relevance of this to my original question. > Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur. Or this, for that matter --- So that you have it, my original ask was: But what's the end user a victim of in this case? Other than a brief lack of availability?
canadianfella 5y ago
Should you answer your own questions? No.
rashkov 5y ago
that first issue links to a really great flow-chart of how iptables works: https://cesarti.files.wordpress.com/2012/02/iptables.gif https://cesarti.files.wordpress.com/2012/02/iptables.gif
2ion 5y ago
> It's not all that great blaming the victim * Run a SaaS that costs money * Do not perform the migration in prod replica setup with test data and monitor for, observe and fix any oddities * Cause a 100% data breach by not following a sound change management procedure Just because this individual is being blamed and not a Ltd. or Corp. doesn't mean that this is how a quality SaaS should be run. I know that more it's-just-a-side-gig SaaS wing it more than they do not but this is exactly what happens when you haven't got good ops. Rather than random error, this is a consequence from a systematic error.
gentleman11 5y ago
I haven’t used docker in production. I guess it is used for k8s all around?

Firerouge 5y ago

This same one got me after years of using Docker, I only discovered it after using the combination of Ubuntu Server (and it's ufw) on a DMZed device. I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords. I've been using https://github.com/shinebayar-g/ufw-docker-automated https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers. It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.

cbo100 5y ago
Yep. This exact interaction between UFW and Docker had me stumped for a while too. Totally unexpected outcome. Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.

1337shadow 5y ago

> Published ports > This creates a firewall rule which maps a container port to a port on the Docker host to the outside world. Source : https://docs.docker.com/config/containers/container-networking/ https://docs.docker.com/config/containers/container-networki... Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost". However testing your firewall after publishing a docker port seems common sense. Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/ And that's how I made a simple firewall that works: https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/main.yml#L77-115 https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/... Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it. So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017. But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".

411111111111111 5y ago
> Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. That only works with local communication unless you use docker swarm. So byebye high availability. If you're going to make suggestions, at least think about them from a production point of view.
- 1337shadow 5y ago
  Most productions are fine with 99.9% uptime, and a single server is fine for that, so HA is not necessary for most production out there. For example, I'm running a governmental, nationwide service handling 300k req/day, with 1k admins working daily on the site, on a single server without anyone complaining, and without less than 99.9% uptime even though I reboot to upgrade the kernel twice a month, I did have to fine tune the system but that was actually pretty easy.
  - yoz-y 5y ago
    That’s 10qps assuming working hours indeed you don’t need more than one server for that.
    - 1337shadow 5y ago
      How many user facing projects actually got so far they needed more than one server? 1% of them?
      - bildung 5y ago
        I'd say even less - a single db server is almost enough for let's encrypt: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...
  - hyperman1 5y ago

nuker 5y ago

> They clearly made the right moves The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT. Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"

diamondo25 5y ago
Even Google Cloud SQL just is a managed instance with a public IP (that you don't seem to be able to make private). Firewalls always help, and thats what Cloud SQL does: whitelist your ip when using the CLI.
- nuker 5y ago
  What I do in AWS is having a VPC with 3 subnet types, public, private and restricted. Public ones have default route to inet, private to NAT, restricted - none. DBs go in restricted, load balancers in public and web/app servers in private subnets. Plus whitelist firewalling using Security Groups.
  - mercora 5y ago
    when i evaluated using a NAT setup using AWS NAT gateways it appeared to be quite expensive. we settled for strict VPC rules using only SecurityGroups in order to restrict communication. the instances did have public ip but nothing but intended services could connect to it anyways... i.e an instance running a HTTP API could only be connected to if initiated from the load balancer and itself could only connect to relevant services such as a RDS cluster...
    - nuker 5y ago
      Normally machines behind NAT use it only for pulling stuff on boot time. Main traffic goes via LBs. depends on your case, of course.
      - 5y ago

mschuster91 5y ago

> Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion. It's clearly written in the docs: To expose a container’s internal port, an operator can start the container with the -P or -p flag. The exposed port is accessible on the host and the ports are available to any client that can reach the host. (from https://docs.docker.com/engine/reference/run/#expose-incoming-ports https://docs.docker.com/engine/reference/run/#expose-incomin...) It should be common knowledge by now to either create a virtual interface / network on the host to publish ports on for usage by services that aren't running in Docker or (if you are in an environment where all services are in Docker) use --link between containers.

arp242 5y ago
I don't think that's clear at all. If I set "bind_ip = *" in some application then it's also "available to any client that can reach the host", but the firewall is in front of that. I certainly wouldn't expect an application to frob with my firewall. And as I understand it, this is very much an unintentional side-effect of ufw and Docker interacting – it's not Docker's the intention at all to override any iptable rules, just an unfortunate side-effect. "It should be common knowledge by now" is very hand-wavy. I never used Docker much, and I could have been bitten by this. Where do I get this "common knowledge" from? Not everyone is a full-time sysadmin; some people just want to run a small service, read the basic documentation on various things etc., and set things up. Not everyone is super-invested in Docker (or any other tool for that matter) to have all the "common knowledge" that all the experts have. This is why defaults matter.
- secabeen 5y ago
  Same here. It's even a little worse because of the vagaries of Docker networking. I have a server, it has a firewall that has all ports blocked. If I install Docker to run some containers and use the --network=host option, it all works fine. Containers listen on ports on the host, my existing firewall controls it, great. If I use what would seem like the more secure port-forward option, Docker adds a new rule to the top of my firewall ruleset opening the forwarded ports to all traffic, overriding my existing firewall for that port entirely. I get that Docker is often run by devs on their laptops, but this is bad behavior. The simplest solution I've seen so-far is Docker should only automatically set these firewall rules on devices that only have RFC1918 IPs. That would still keep things simple for devs on laptops, but would prevent firewall override on servers.

bildung 5y ago

> It's not all that great blaming the victim. Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management https://www.sebokwiki.org/wiki/Risk_Management

eganist 5y ago
> Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. There can be multiple victims, multiple causes/threat actors, and overlap between two categories. But what's the end user a victim of in this case? Other than a brief lack of availability?
- bildung 5y ago
  Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring? Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
  - eganist 5y ago
    > Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring? I don't understand the relevance of this to my original question. > Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur. Or this, for that matter --- So that you have it, my original ask was: But what's the end user a victim of in this case? Other than a brief lack of availability?

canadianfella 5y ago

Should you answer your own questions? No.

rashkov 5y ago

that first issue links to a really great flow-chart of how iptables works: https://cesarti.files.wordpress.com/2012/02/iptables.gif https://cesarti.files.wordpress.com/2012/02/iptables.gif

2ion 5y ago

> It's not all that great blaming the victim * Run a SaaS that costs money * Do not perform the migration in prod replica setup with test data and monitor for, observe and fix any oddities * Cause a 100% data breach by not following a sound change management procedure Just because this individual is being blamed and not a Ltd. or Corp. doesn't mean that this is how a quality SaaS should be run. I know that more it's-just-a-side-gig SaaS wing it more than they do not but this is exactly what happens when you haven't got good ops. Rather than random error, this is a consequence from a systematic error.

gentleman11 5y ago

I haven’t used docker in production. I guess it is used for k8s all around?