5 ms·

As a lowly user¹: linux containers are more like gaffer tape around namespaces and cgroups than something like lego. You want real memory usage in your cgroup?

• djsumdog 10y ago
  I really wish this post went into more detail. It feels too high level to be useful. I ran into the memory issue recently. In DC/OS when you use the marathon scheduler, if you go above the allocated memory limit, the scheduler kills your task and restarts it. The trouble is, if you ran top inside your container, and you're running on a DC/OS node with 32GB of memory, top reports all 32GB of memory. So interpreters that do garbage collection (like Java) will just continue to eat memory if you don't specify the right limits/parameters. The OS will eve let it allocate past the container limit, but just kill the container afterwards. Now the container limit is available under the /proc/cgroups somewhere, but now interpreters need to check to see if they're running in a container and adjust everything accordingly. Of course you could always tell your scheduler not to hard kill something when it goes over a memory limit, which is why we never ran into that when we were running things on CoreOS since we didn't configure hard limits per container.
  • Xylakant 10y ago
    > top reports all 32GB of memory. Same applies to CPU and number of cores. A common pitfall for example for ElasticSearch, which bases its default threadpools on the number of visible CPUs. The isolation layer is indeed very thing and leaky in all places.
    • XorNot 10y ago
      Is there a common kernel patch dealing with this problem yet? I know it's possible, but Google is light on links and it's a problem I have right now.
    • djsumdog 10y ago
      But with CPU cores, it's not a hard limit. At least in Marathon, you can ask for 0.5 CPUs and marathon will use that to decide where your container gets scheduled (so if it has 16 cores, all scheduled containers should add up to 16; which is why you should use fractions of a core to start and then monitor to see real CPU usage) , but the container gets all of the CPUs. I don't know if that's specific to our Marathon/DCOS setup or if that's a limitation of the underlying Docker daemon. Not sure how K8s or Nomad handle CPU/memory limits either.
      • schmichael 10y ago
        Nomad's exec & java drivers handle CPU and memory through cgroups (via libcontainer). In particular CPU uses CPU shares[0] so all cores/CPUs are treated as a single pool of shares. In the future we plan on adding CPU pinning and might add cores as a unit of CPU resources. The idea of constraining CPU resources by core is especially important on ARM servers which don't always report their clockspeed. Packet.net for example has 96 core ARM boxes, so users would probably prefer to set how many cores they need instead of MHz. [0] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
  • chatmasta 10y ago
    This is an interesting observation. It seems the simple fix would be tricking the containerized application into reading the "total system memory" from a syscall hooked by the container runtime to return the configured memory limit of the container making the syscall. I'm surprised this is not already done; is there some inherent limit that prohibits this? It seems an unintended consequence of containerizaion is that the responsibility for garbage collection effectively moves from the containerized application (e.g. the interpreter) to the container runtime, which "collects garbage" by terminating containers at their memory limit, just like a process level garbage collector terminates (or prunes) functions or data structures at their memory limit. I'm not sure this is a bad thing. Moving garbage collection up one level in the "stack" of abstraction seems in line with the idea that containers are the building blocks of a "data center operating system." Naturally then, shouldn't garbage collection happen at the level of the container runtime? Otherwise you're wasting compute cycles by collecting garbage at two levels. When garbage collection moves to the container runtime, it should mean that the application no longer has to worry about garbage collection, since the container runtime will terminate the container when it reaches its memory limit. Therefore, the application (e.g. the java interpreter) only needs to make sure it can handle frequent restarts. In practice this means coding stateless applications with fast startup times. Applications like the java interpreter were designed in an era dominated by long running, stateful processes. Now we are seeing a move to stateless applications with fast boot times (i.e. "serverless" shudder). Stateless applications are a prerequisite to turning the data center into an "operating system" because they essentially take the role of function calls in a traditional operating system. Both containers and function calls are the "building blocks" of their respective levels of abstraction. In a traditional OS, you wouldn't expect a single function call to run forever and do its own garbage collection, so why would you expect the same from a containerized application in a datacenter OS?
• zuzun 10y ago
  Yeah, Linux security features work like: throw ... against a wall and see what sticks. I find it amusing when people say: "We should write a new kernel" and their only proposed security feature is using that memory safe language (TM)... they'd have my attention if they said "We should write a new kernel and design all the permissions/isolations/resource limits from the ground up".
  • dom0 10y ago
    I.e. an enterprise operating system.
• erikb 10y ago
  Yes, and as long as your life isn't threatened and you live in a world full of other people, problems, opportunities, the lego ship with gaffa tape is way more useful.