4 ms·

Container approach is not always feasible, for example when you need to stay on bare metal. There are Linux applications where selinux definitely gets in your w

Container approach is not always feasible, for example when you need to stay on bare metal. There are Linux applications where selinux definitely gets in your way and there are legitimate reasons to turn it off completely. edit: I like how I'm getting down-voted for sharing factual information. No-one in high frequency trading employs containers because it doesn't help soft-realtime, low-latency goals. There are other performance-sensitive domains where containers aren't an option. Sometimes I feel like this place is turning into Reddit.

TheDong 5y ago
> when you need to stay on bare metal What cases can linux containers not handle? Containers can access GPUs, /dev/kvm, block devices... I'm having trouble thinking of anything they can't do. After all, they're just processes in a glorified chroot, not that different from processes on the linux host.
- Gigachad 5y ago
  Not sure about need but I do remember it being a pain in the ass to run anything that expects to be managing docker inside docker. Things like self hosted gitlab ci where its trying to start up containers itself.
  - KronisLV 5y ago
    > ...anything that expects to be managing docker inside docker. Now that's an interesting problem to have! If you trust those tools and don't have untrusted users or untrusted code, you can sometimes just mount /var/run/docker.sock and use the VM/VPS/server's Docker directly. It is actually the approach that was used by excellent tools like Portainer, though it's also really risky as well. Alternatively, you can try to just run Docker in Docker (DinD), which is a bit more tricky and the opinions there are split about whether to do it and when to do it: https://github.com/jpetazzo/dind https://github.com/jpetazzo/dind Of course, someone might also jump in and suggest that Docker is architecturally problematic (i don't care much, just want my containers to run, then again; i don't deal with untrusted code or any sort of multitenancy) and you should use Podman or another set of technologies, which is interesting advice but would necessitate other approaches. In short, like with most technologies: Docker and OCI container in general get more messy as your requirements become more advanced. For the problems that they do solve easily (app packaging), they are pretty good, though!
    - Koffiepoeder 5y ago
      One project in this space that looked quite promising to me is sysbox[0]. I've used them once for a gitlab runner set-up similar to what is described in their blog[1]. It's currently working great and I have not had any major crashes/incidents for at least the past 8 months. [0]: https://github.com/nestybox/sysbox https://github.com/nestybox/sysbox [1]: https://blog.nestybox.com/2020/10/21/gitlab-dind.html https://blog.nestybox.com/2020/10/21/gitlab-dind.html
  - johannes1234321 5y ago
    Mounting the docker socket into a container works. If it's advisable is another question, as that gives full control over all other containers (and likely more)
    - staticassertion 5y ago
      Mounting the docker socket makes it trivial to escape to the host since you can just launch a privileged container or mount arbitrary files. Plus the daemon generally runs as root. At that point you may want to just run it on the host, but basically you'd better trust whatever it is.
- bitcharmer 5y ago
  Ultra-low latency applications on Linux are definitely one domain were containers are a big no-no for performance reasons. In my field (HFT) you won't find any serious performance-sensitive deployments using containers. So that's one example.
  - megous 5y ago
    Why would a container increase latency? It's just a pointer on a process. It's there on all processes. https://elixir.bootlin.com/linux/latest/source/include/linux/sched.h#L1080 https://elixir.bootlin.com/linux/latest/source/include/linux...
    - geraldwhen 5y ago
      https://www.diva-portal.org/smash/get/diva2:1252694/FULLTEXT01.pdf https://www.diva-portal.org/smash/get/diva2:1252694/FULLTEXT... Also docker NAT will add significant latency if you don’t use host network mode. When microseconds matter, everything matters. Any intermediate program would be suspect.
      - megous 5y ago
        Then that's not about containers, but about NAT. So maybe don't use NAT. You can have NAT on the host, too, and it would slow down things also. "Container" (namespace to be precise) is not really an intermediate program, it's a state associated with any Linux process.
  - CraigJPerry 5y ago
    Are you referring to kube rather than containers? A container is just namespaces and cgroups. You can still have your mellanox or solarflare zero-copy network stack for a containerised process. That is, a containerised user space process can directly access the hardware, bypassing the kernel but isolated from other processes. There are plenty of footguns for sure though but my experience working in HFT has been everyone is pretty elite skill-wise so sharp edges are not an issue in practice.
- gjulianm 5y ago
  I've had quite some problems setting up DPDK and other device drivers inside of containers. It looks like it should be easy, yes, but then reality hits you in the face.

TheDong 5y ago

> when you need to stay on bare metal What cases can linux containers not handle? Containers can access GPUs, /dev/kvm, block devices... I'm having trouble thinking of anything they can't do. After all, they're just processes in a glorified chroot, not that different from processes on the linux host.

Gigachad 5y ago
Not sure about need but I do remember it being a pain in the ass to run anything that expects to be managing docker inside docker. Things like self hosted gitlab ci where its trying to start up containers itself.
- KronisLV 5y ago
  > ...anything that expects to be managing docker inside docker. Now that's an interesting problem to have! If you trust those tools and don't have untrusted users or untrusted code, you can sometimes just mount /var/run/docker.sock and use the VM/VPS/server's Docker directly. It is actually the approach that was used by excellent tools like Portainer, though it's also really risky as well. Alternatively, you can try to just run Docker in Docker (DinD), which is a bit more tricky and the opinions there are split about whether to do it and when to do it: https://github.com/jpetazzo/dind https://github.com/jpetazzo/dind Of course, someone might also jump in and suggest that Docker is architecturally problematic (i don't care much, just want my containers to run, then again; i don't deal with untrusted code or any sort of multitenancy) and you should use Podman or another set of technologies, which is interesting advice but would necessitate other approaches. In short, like with most technologies: Docker and OCI container in general get more messy as your requirements become more advanced. For the problems that they do solve easily (app packaging), they are pretty good, though!
  - Koffiepoeder 5y ago
    One project in this space that looked quite promising to me is sysbox[0]. I've used them once for a gitlab runner set-up similar to what is described in their blog[1]. It's currently working great and I have not had any major crashes/incidents for at least the past 8 months. [0]: https://github.com/nestybox/sysbox https://github.com/nestybox/sysbox [1]: https://blog.nestybox.com/2020/10/21/gitlab-dind.html https://blog.nestybox.com/2020/10/21/gitlab-dind.html
- johannes1234321 5y ago
  Mounting the docker socket into a container works. If it's advisable is another question, as that gives full control over all other containers (and likely more)
  - staticassertion 5y ago
    Mounting the docker socket makes it trivial to escape to the host since you can just launch a privileged container or mount arbitrary files. Plus the daemon generally runs as root. At that point you may want to just run it on the host, but basically you'd better trust whatever it is.
bitcharmer 5y ago
Ultra-low latency applications on Linux are definitely one domain were containers are a big no-no for performance reasons. In my field (HFT) you won't find any serious performance-sensitive deployments using containers. So that's one example.
- megous 5y ago
  Why would a container increase latency? It's just a pointer on a process. It's there on all processes. https://elixir.bootlin.com/linux/latest/source/include/linux/sched.h#L1080 https://elixir.bootlin.com/linux/latest/source/include/linux...
  - geraldwhen 5y ago
    https://www.diva-portal.org/smash/get/diva2:1252694/FULLTEXT01.pdf https://www.diva-portal.org/smash/get/diva2:1252694/FULLTEXT... Also docker NAT will add significant latency if you don’t use host network mode. When microseconds matter, everything matters. Any intermediate program would be suspect.
    - megous 5y ago
      Then that's not about containers, but about NAT. So maybe don't use NAT. You can have NAT on the host, too, and it would slow down things also. "Container" (namespace to be precise) is not really an intermediate program, it's a state associated with any Linux process.
- CraigJPerry 5y ago
  Are you referring to kube rather than containers? A container is just namespaces and cgroups. You can still have your mellanox or solarflare zero-copy network stack for a containerised process. That is, a containerised user space process can directly access the hardware, bypassing the kernel but isolated from other processes. There are plenty of footguns for sure though but my experience working in HFT has been everyone is pretty elite skill-wise so sharp edges are not an issue in practice.
gjulianm 5y ago
I've had quite some problems setting up DPDK and other device drivers inside of containers. It looks like it should be easy, yes, but then reality hits you in the face.