5 ms·

And don't forget to include making sure to create the right resource and namespace constraints so that the one binary doesn't gobble up memory endlessly or use

And don't forget to include making sure to create the right resource and namespace constraints so that the one binary doesn't gobble up memory endlessly or use up the file system or put your infrastructure completely at risk if/when it's compromised -- whether you do that with systemd units, raw cgroup/namespace finagling, or regular tried-and-true linux user-based resource segregation. Containers do provide benefits.

SahAssar 6y ago
So ship the static binary and a systemd unit file like a lot of packages in the repos of your linux distro do.
- hardwaresofton 6y ago
  Yep, and you'd generally expect to spend time understanding the complexity that systemd brings, the subsystems that power it, and how you can separate resources there! There is no free lunch. Restated, my point here is that while containers can be more complex and have pitfalls (a lot of which have been worked out somewhat at this point), there is no complexity free lunch -- `[docker|podman|crictl] run --rm --cpus 2 --memory 500mb ...` is pretty darn easy, and more so than writing properly portable and well-considered systemd unit files (and putting them in the right place, with the right permissions, under the right slice, etc). It's easier than most of the options out there (including the old methods of per-user resource segregation).
  - c0l0 6y ago
    I don't see how appending --cpus 2 --memory 500mb to the argument vector liberates these options' definer from "understanding [...] the subsystems that power it", and reflect on their potential impact. And I'd argue that THIS is the real complexity incurred by these kinds of resource constraints that seem so simple on the surface - not the specific syntax or location you have to use to introduce them. All of which makes systemd unit files and their settings' implications (which are amazingly well-documented btw) as good as any other option, imho.
    - hardwaresofton 6y ago
      Systemd is great software -- it is useful, and powerful. A systemd unit file, the idea of what a unit is, when they run, how they run, what shells they use, what permissions they run under, and lots of other complexities are expressed as options in the unit file, configuration files on the system, and in other places. If I want to run a useful piece of software like redis let's say, but I want to run it with a resource constraint to make sure that it doesn't take more than 2CPUs and 500MB of memory, it is far easier to do that with the following command line: docker run --rm redis --cpus 2 --memory 500mb -p 6379:6379 Than to write the equivalent systemd unit file, set up the isolated filesystems that docker would let you easily bind mount in, etc. This is like comparing systemd-nspawn to systemd -- if systemd-nspawn isn't simpler than systemd then what are we even doing. Docker won because of it's developer ergonomics (containers weren't new), systemd won because of it's feature set, convenience and sturdiness. They're different tools with different primary use-cases.
      - hartmel 6y ago
        Then you will want to manage the configuration file and to browse logs, and you ll have to mount files to you container. And your deployment start to reimplement (a subset to your needs) what distros did System integration complexity didn't disappear thanks to containers. It was just moved to another place. I dd add that, being convenient for developers doesn't mean it's convenient for hosting management. At the end, when one side convenience is taken in consideration, it often translate to complexity and pain at the other side. K8S become in some companies more a mandatory runtime requirement than a hosting commodity/facility. As a (former) developer I d rather ran redis in a container during development, but for production I d rather rely on boring VMs unless some scaling is required. (Managed k8s case set apart)
        hardwaresofton 6y ago
        > Then you will want to manage the configuration file and to browse logs, and you ll have to mount files to you container. And your deployment start to reimplement (a subset to your needs) what distros did System integration complexity didn't disappear thanks to containers. Agree, but my view on this is that the implicit answer of how you do all that (i.e. the file system, syslog) is now gone. There will be pain (complexity) in the short term, but at the end of the day, we're going to be able to build much better orchestration and systems. To kind of restate that, before you had to worry where a process wrote out it's output (stdout? /var/log/<program>? /etc/<program>/logs? /home/<user>/<program>/logs? syslog?), now you know want to get the non-stdout/stderr logs of the thing you're running, you'd better give it a volume to write to (which may be fake, and actually write everything to some remote storage or something), and I think that's a step forward. Of course, I'm not saying containers should go everywhere -- relying on boring VMs over containers is fine too -- but I think rich world of functionality available to container-driven workflows is popular for good and bad reasons, and the good reasons are worth exploring/beneficial to me.
  - SahAssar 6y ago
    Let's say you are installing this from a normal package manager and want to limit it: To install: apt install mypackage To edit the systemd unit (automatically creating the file in the right place): systemctl edit mypackage Then add these lines to limit to 2 cpus and 500mb memory: [Service] CPUAccounting=true CPUQuota=200% MemoryAccounting=true MemoryHigh=500M And then to start it now and on boot: systemctl enable --now mypackage This is now integrated with package updates, starts on boot, logs to the same place that most other system utilities do and so on.
    - hardwaresofton 6y ago
      I'm not about to argue against systemd, it's great software, and it's in every distro right now for a reason. Understanding apt (+/- how to package for it), systemctl, and the options you've laid out in your unit file are not trivial, and I would argue that they are less trivial (or harder) than understanding what's happening with containers, especially if you're running rootless containers, and/or using a container tool like podman which does without the daemon. --cpus 2 --memory 500M is easier, and gets you the same results though they may not be as permanent or as well managed -- the management and external stuff is an orthogonal concern, and that's not the situation I was addressing. The original point was insinuating that throwing up a binary and getting it to run. your filesystem is also not available to the container by default, and in this way docker sort of fails closed. If you're running a rootless container, the story is even better. One thing you have not covered is filesystem isolation, which docker also does very easily. There is a lot to configure on the systemd side[0] and the parts that are overlapping are just easier to configure and run with docker. Systemd is the better tool to build repeatable installs for pet processes, but again, there is a lot of knowledge underneath that is related. People to this day still complain that systemd does too much (I personally like it a lot, and it's great to have everything in one place). [EDIT] Just to make myself clear, systemd is an amazing tool -- I like it, I run it, I'm not smart enough to administer a more complicated setup -- but docker is easier, for a large part of the small subset of systemd's capabilities that docker covers. [0]: https://www.redhat.com/sysadmin/mastering-systemd https://www.redhat.com/sysadmin/mastering-systemd
    - aequitas 6y ago
- jeroenhd 6y ago
  I don't understand why this isn't being done more. systemd has pretty much every convenience Docker has these days, without any of the ridiculous overhead and cognitive load that comes with managing a Docker daemon. Yes, you'd need to learn how to make a systemd unit file but honestly, that shouldn't be a problem at all. I've had much worse headaches fighting Docker's networks and firewall-overruling network configuration in the past. Every time I forget to specify a restart option, I have to dig through my shell history again to see how I launched the image so I can kill and recreate it, or I have to look up that oneliner that shows you the docker run command for a running container. I run stuff in Docker for one simple reason: I have it installed and I'm too lazy to think about software sometimes. If you're packaging software, that laziness isn't a reason to pick a distribution tool. Static binaries have other problems, perhaps most importantly package management. How do you distribute your app? Traditional deb/rpm packages? curl-to-bash installers? Snap? Flatpak? Binaries that people flog into /usr/bin? What about automated updates, do you set up a repository, do you include a self-updater in your code? The list goes on. Docker solves that by having one general source of packages with the option of adding a repo of your own (take note, Canonical, your shitty Snap Store is practically useless without that last bit). For services I run on servers, I much prefer traditional packages with dynamically linked executables, for a simple reason: automatically fixing security issues and bugs across applications with a single update command, instead of having to wait for every maintainer of their statically-linked tool to update their dependencies. It's one of the big problems I have with Rust; many dependencies and applications embed their own versions of dependencies with sometimes very specific versions, and when there will eventually be a massive security problem in one of the TLS packages I'm dependent on recompiles and code modifications from random open source maintainers.
  - goodpoint 6y ago
    > I don't understand why this isn't being done more. I can only offer one explanation: marketing.
  - dolmen 6y ago
    Here, we use docker-compose to automate container creation. And we have just one Makefile to automate docker-compose (to fetch credentials from secrets storage). Our images are made from Go binaries, so thanks to Go modules, we can manage dependencies centrally from the main module and rebuild.
IshKebab 6y ago
Most servers are already running in a VM that takes care of all that.
- hardwaresofton 6y ago
  yeah but that bit of complexity wasn't mentioned :). If every process you need to run in it's own VMs/per-instance then you've got best-in-class isolation already, no need to bother with containers, and you can get your resource and namespace isolation. Not quite the same as just throwing a static binary over the wall though, a little bit more complexity there.
krageon 6y ago
> compromised Containers are not a security measure. They're acceptable for some resource constraint considerations, but you should not ever use them because of security concerns. If that's something to worry about, use VMs at the very least.
- hardwaresofton 6y ago
  > Containers are not a security measure. They're acceptable for some resource constraint considerations, but you should not ever use them because of security concerns. If that's something to worry about, use VMs at the very least. This is repeatedly really commonly, but is bordering on wrong more and more every year. At the lowest level it's not even completely correct. Containers are resource + namespace isolation -- cgroups and namespaces -- isolation and kernel-assisted resource hiding/access control is absolutely a facet of good defence-in-depth. Enhanced security is not a core feature of containerization as a whole but it can absolutely help with your security posture. For what % that satement is wrong, it's even more so because it fails to consider the container runtime. Container runtimes these days (containerd is the best one IMO) include ways to run containers at different levels of isolation (VMs and as micro kernels), so a move to containers can absolutely help security posture, because now you don't have to pull out packer/ansible/etc to build your VM, you can just take the same container you were running and change the runtime. Time to ship/launch improved security features is important, and containers can reduce that burden for teams. Containers are a process isolation and resource control tool -- they can absolutely aid in security posture. Would you say the same about BSD Jails? Just so there's some more value here, here are some projects that are somewhat close to cutting edge (though it's been a while) in this space that I like: - nabla-containers.github.io/ - https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... - https://github.com/kata-containers/runtime https://github.com/kata-containers/runtime (see https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md https://github.com/kata-containers/documentation/blob/master...) - https://rootlesscontaine.rs/ https://rootlesscontaine.rs/ - https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2... - https://starkandwayne.com/blog/the-capable-kernel-an-introduction-to-linux-capabilities/ https://starkandwayne.com/blog/the-capable-kernel-an-introdu... The container ecosystem surfaces a lot of really cool resource limiting and isolation techniques and puts them in very close reach.
pjmlp 6y ago
My first container experience goes back to HP-UX Vaults. I let you research when that was, and wonder what container technologies might have used since then. I know when they make sense and when they happen to be fashion.
- hardwaresofton 6y ago
  Didn't doubt your experience (I read and enjoy your comments all the time on HN), just pointed out that the feature set offered by single static binary (which arguably is not easy to build depending on what language you're using, etc) is not the same as what's offered by linux containerization in the current day. Half the "static" binaries that used to float around weren't even fully static because of glibc, until building with musl (and getting bit by getaddrinfo) became widespread -- the situation is not as simple as you made it seem.
  - pjmlp 6y ago
    I agree with the glibc issue, but that is a problem of GCC specifically. My point is that I see containers being overused nowadays, like "big data" that fits on a USB pen kind of scenario.
    - hardwaresofton 6y ago
      Agreed -- containers are certainly being overused these days, I find a big indicator of when people know what they're talking about is when they mention/realize that docker is containerd these days (docker is simply a shim over containerd which uses runc underneath), but it is absolutely still very buzzword-y. I think we've basically stumbled into a very effective and widespread packaging paradigm though -- now you don't even have to pick the right language/toolset to get static binaries easily -- just throw a container over the wall and your filesystem requirements (mounts), network requirements, etc will be made pretty obvious to the person doing the deployment
      - pjmlp 6y ago
        Assuming that source and target hardware are the same, otherwise VMs enter into the picture, and stuff like Kata Containers add even more layers. Then I remember Linus point of view about monolitic kernels and how they are supposed to beat micro-kernels, what for, when people put hundreds of virtualization layers on top.
ex_amazon_sde 6y ago
You are describing sandboxing and it's completely orthogonal to containers.

SahAssar 6y ago

So ship the static binary and a systemd unit file like a lot of packages in the repos of your linux distro do.

hardwaresofton 6y ago
Yep, and you'd generally expect to spend time understanding the complexity that systemd brings, the subsystems that power it, and how you can separate resources there! There is no free lunch. Restated, my point here is that while containers can be more complex and have pitfalls (a lot of which have been worked out somewhat at this point), there is no complexity free lunch -- `[docker|podman|crictl] run --rm --cpus 2 --memory 500mb ...` is pretty darn easy, and more so than writing properly portable and well-considered systemd unit files (and putting them in the right place, with the right permissions, under the right slice, etc). It's easier than most of the options out there (including the old methods of per-user resource segregation).
- c0l0 6y ago
  I don't see how appending --cpus 2 --memory 500mb to the argument vector liberates these options' definer from "understanding [...] the subsystems that power it", and reflect on their potential impact. And I'd argue that THIS is the real complexity incurred by these kinds of resource constraints that seem so simple on the surface - not the specific syntax or location you have to use to introduce them. All of which makes systemd unit files and their settings' implications (which are amazingly well-documented btw) as good as any other option, imho.
  - hardwaresofton 6y ago
    Systemd is great software -- it is useful, and powerful. A systemd unit file, the idea of what a unit is, when they run, how they run, what shells they use, what permissions they run under, and lots of other complexities are expressed as options in the unit file, configuration files on the system, and in other places. If I want to run a useful piece of software like redis let's say, but I want to run it with a resource constraint to make sure that it doesn't take more than 2CPUs and 500MB of memory, it is far easier to do that with the following command line: docker run --rm redis --cpus 2 --memory 500mb -p 6379:6379 Than to write the equivalent systemd unit file, set up the isolated filesystems that docker would let you easily bind mount in, etc. This is like comparing systemd-nspawn to systemd -- if systemd-nspawn isn't simpler than systemd then what are we even doing. Docker won because of it's developer ergonomics (containers weren't new), systemd won because of it's feature set, convenience and sturdiness. They're different tools with different primary use-cases.
    - hartmel 6y ago
      Then you will want to manage the configuration file and to browse logs, and you ll have to mount files to you container. And your deployment start to reimplement (a subset to your needs) what distros did System integration complexity didn't disappear thanks to containers. It was just moved to another place. I dd add that, being convenient for developers doesn't mean it's convenient for hosting management. At the end, when one side convenience is taken in consideration, it often translate to complexity and pain at the other side. K8S become in some companies more a mandatory runtime requirement than a hosting commodity/facility. As a (former) developer I d rather ran redis in a container during development, but for production I d rather rely on boring VMs unless some scaling is required. (Managed k8s case set apart)
      - hardwaresofton 6y ago
        > Then you will want to manage the configuration file and to browse logs, and you ll have to mount files to you container. And your deployment start to reimplement (a subset to your needs) what distros did System integration complexity didn't disappear thanks to containers. Agree, but my view on this is that the implicit answer of how you do all that (i.e. the file system, syslog) is now gone. There will be pain (complexity) in the short term, but at the end of the day, we're going to be able to build much better orchestration and systems. To kind of restate that, before you had to worry where a process wrote out it's output (stdout? /var/log/<program>? /etc/<program>/logs? /home/<user>/<program>/logs? syslog?), now you know want to get the non-stdout/stderr logs of the thing you're running, you'd better give it a volume to write to (which may be fake, and actually write everything to some remote storage or something), and I think that's a step forward. Of course, I'm not saying containers should go everywhere -- relying on boring VMs over containers is fine too -- but I think rich world of functionality available to container-driven workflows is popular for good and bad reasons, and the good reasons are worth exploring/beneficial to me.
- SahAssar 6y ago
  Let's say you are installing this from a normal package manager and want to limit it: To install: apt install mypackage To edit the systemd unit (automatically creating the file in the right place): systemctl edit mypackage Then add these lines to limit to 2 cpus and 500mb memory: [Service] CPUAccounting=true CPUQuota=200% MemoryAccounting=true MemoryHigh=500M And then to start it now and on boot: systemctl enable --now mypackage This is now integrated with package updates, starts on boot, logs to the same place that most other system utilities do and so on.
  - hardwaresofton 6y ago
    I'm not about to argue against systemd, it's great software, and it's in every distro right now for a reason. Understanding apt (+/- how to package for it), systemctl, and the options you've laid out in your unit file are not trivial, and I would argue that they are less trivial (or harder) than understanding what's happening with containers, especially if you're running rootless containers, and/or using a container tool like podman which does without the daemon. --cpus 2 --memory 500M is easier, and gets you the same results though they may not be as permanent or as well managed -- the management and external stuff is an orthogonal concern, and that's not the situation I was addressing. The original point was insinuating that throwing up a binary and getting it to run. your filesystem is also not available to the container by default, and in this way docker sort of fails closed. If you're running a rootless container, the story is even better. One thing you have not covered is filesystem isolation, which docker also does very easily. There is a lot to configure on the systemd side[0] and the parts that are overlapping are just easier to configure and run with docker. Systemd is the better tool to build repeatable installs for pet processes, but again, there is a lot of knowledge underneath that is related. People to this day still complain that systemd does too much (I personally like it a lot, and it's great to have everything in one place). [EDIT] Just to make myself clear, systemd is an amazing tool -- I like it, I run it, I'm not smart enough to administer a more complicated setup -- but docker is easier, for a large part of the small subset of systemd's capabilities that docker covers. [0]: https://www.redhat.com/sysadmin/mastering-systemd https://www.redhat.com/sysadmin/mastering-systemd
  - aequitas 6y ago
jeroenhd 6y ago
I don't understand why this isn't being done more. systemd has pretty much every convenience Docker has these days, without any of the ridiculous overhead and cognitive load that comes with managing a Docker daemon. Yes, you'd need to learn how to make a systemd unit file but honestly, that shouldn't be a problem at all. I've had much worse headaches fighting Docker's networks and firewall-overruling network configuration in the past. Every time I forget to specify a restart option, I have to dig through my shell history again to see how I launched the image so I can kill and recreate it, or I have to look up that oneliner that shows you the docker run command for a running container. I run stuff in Docker for one simple reason: I have it installed and I'm too lazy to think about software sometimes. If you're packaging software, that laziness isn't a reason to pick a distribution tool. Static binaries have other problems, perhaps most importantly package management. How do you distribute your app? Traditional deb/rpm packages? curl-to-bash installers? Snap? Flatpak? Binaries that people flog into /usr/bin? What about automated updates, do you set up a repository, do you include a self-updater in your code? The list goes on. Docker solves that by having one general source of packages with the option of adding a repo of your own (take note, Canonical, your shitty Snap Store is practically useless without that last bit). For services I run on servers, I much prefer traditional packages with dynamically linked executables, for a simple reason: automatically fixing security issues and bugs across applications with a single update command, instead of having to wait for every maintainer of their statically-linked tool to update their dependencies. It's one of the big problems I have with Rust; many dependencies and applications embed their own versions of dependencies with sometimes very specific versions, and when there will eventually be a massive security problem in one of the TLS packages I'm dependent on recompiles and code modifications from random open source maintainers.
- goodpoint 6y ago
  > I don't understand why this isn't being done more. I can only offer one explanation: marketing.
- dolmen 6y ago
  Here, we use docker-compose to automate container creation. And we have just one Makefile to automate docker-compose (to fetch credentials from secrets storage). Our images are made from Go binaries, so thanks to Go modules, we can manage dependencies centrally from the main module and rebuild.

IshKebab 6y ago

Most servers are already running in a VM that takes care of all that.

hardwaresofton 6y ago
yeah but that bit of complexity wasn't mentioned :). If every process you need to run in it's own VMs/per-instance then you've got best-in-class isolation already, no need to bother with containers, and you can get your resource and namespace isolation. Not quite the same as just throwing a static binary over the wall though, a little bit more complexity there.

krageon 6y ago

> compromised Containers are not a security measure. They're acceptable for some resource constraint considerations, but you should not ever use them because of security concerns. If that's something to worry about, use VMs at the very least.

hardwaresofton 6y ago
> Containers are not a security measure. They're acceptable for some resource constraint considerations, but you should not ever use them because of security concerns. If that's something to worry about, use VMs at the very least. This is repeatedly really commonly, but is bordering on wrong more and more every year. At the lowest level it's not even completely correct. Containers are resource + namespace isolation -- cgroups and namespaces -- isolation and kernel-assisted resource hiding/access control is absolutely a facet of good defence-in-depth. Enhanced security is not a core feature of containerization as a whole but it can absolutely help with your security posture. For what % that satement is wrong, it's even more so because it fails to consider the container runtime. Container runtimes these days (containerd is the best one IMO) include ways to run containers at different levels of isolation (VMs and as micro kernels), so a move to containers can absolutely help security posture, because now you don't have to pull out packer/ansible/etc to build your VM, you can just take the same container you were running and change the runtime. Time to ship/launch improved security features is important, and containers can reduce that burden for teams. Containers are a process isolation and resource control tool -- they can absolutely aid in security posture. Would you say the same about BSD Jails? Just so there's some more value here, here are some projects that are somewhat close to cutting edge (though it's been a while) in this space that I like: - nabla-containers.github.io/ - https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... - https://github.com/kata-containers/runtime https://github.com/kata-containers/runtime (see https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md https://github.com/kata-containers/documentation/blob/master...) - https://rootlesscontaine.rs/ https://rootlesscontaine.rs/ - https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2... - https://starkandwayne.com/blog/the-capable-kernel-an-introduction-to-linux-capabilities/ https://starkandwayne.com/blog/the-capable-kernel-an-introdu... The container ecosystem surfaces a lot of really cool resource limiting and isolation techniques and puts them in very close reach.

pjmlp 6y ago

My first container experience goes back to HP-UX Vaults. I let you research when that was, and wonder what container technologies might have used since then. I know when they make sense and when they happen to be fashion.

hardwaresofton 6y ago
Didn't doubt your experience (I read and enjoy your comments all the time on HN), just pointed out that the feature set offered by single static binary (which arguably is not easy to build depending on what language you're using, etc) is not the same as what's offered by linux containerization in the current day. Half the "static" binaries that used to float around weren't even fully static because of glibc, until building with musl (and getting bit by getaddrinfo) became widespread -- the situation is not as simple as you made it seem.
- pjmlp 6y ago
  I agree with the glibc issue, but that is a problem of GCC specifically. My point is that I see containers being overused nowadays, like "big data" that fits on a USB pen kind of scenario.
  - hardwaresofton 6y ago
    Agreed -- containers are certainly being overused these days, I find a big indicator of when people know what they're talking about is when they mention/realize that docker is containerd these days (docker is simply a shim over containerd which uses runc underneath), but it is absolutely still very buzzword-y. I think we've basically stumbled into a very effective and widespread packaging paradigm though -- now you don't even have to pick the right language/toolset to get static binaries easily -- just throw a container over the wall and your filesystem requirements (mounts), network requirements, etc will be made pretty obvious to the person doing the deployment
    - pjmlp 6y ago
      Assuming that source and target hardware are the same, otherwise VMs enter into the picture, and stuff like Kata Containers add even more layers. Then I remember Linus point of view about monolitic kernels and how they are supposed to beat micro-kernels, what for, when people put hundreds of virtualization layers on top.

ex_amazon_sde 6y ago

You are describing sandboxing and it's completely orthogonal to containers.