5 ms·

The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. Containers don't really sol

The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. Containers don't really solve your problem, it just adds a layer of indirection between the processes hogging RAM and the OOM killer, which in my experience adds even more uncertainty into what the OOM killer is going to do.

VectorLock 7y ago
Tuning oomkiller's hitlist makes it much more managable and the new BPF tools to track it make it a ton nicer than a system grinding into swap.
scottlamb 7y ago
> The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. I won't say there's no circumstance in which the OOM killer is worse, but in general I disagree with you. A stateful server application: maybe a standard ACID DBMS like PostgreSQL. I'd much rather it get killed (failing whatever transactions were in progress) and restart in a healthy state than continue to run slowly. Here I'm relying on a correct implementation of Durability, but if you don't have that, you have other problems. Even more so for a quorum-based database server in which the leader dying (and another taking over) is much better than continuing to run slowly. And yes, even more so in stateless servers, particularly when there are several of them. Going down and restarting quickly (a brief loss of capacity, a brief spike of errors if the client doesn't retry) is a lot better than continuing to be slow (a sustained loss of capacity, possibly leading to cascading overload; and user-visible latency problems if there's no hedging). For something like a desktop application, it's debatable, but personally I'd still rather endure the up-front nuisance of having to restart it than have the OS try to paper over it until I finally realize why it's so painful and do something about it. (Insert anecdote about frog boiling here; I hear frogs actually do jump out before they boil, but yet somehow the story still rings true, if you know what I mean.)
- mchristen 7y ago
  Databases are just one kind of stateful server application though, and yes I would agree with you for that usecase relying on it's underlying durability is a good thing, because it's built with that in mind. Instead I would like to point out the ubiquity of things like nodejs, which make it all to easy for under-experienced developers to shoot themselves in the foot w/ a chaingun with respect to data design and management. It's not really the internal state of the nodejs process itself, it's the external state that it is inevitably being manipulated in the node process.
  - scottlamb 7y ago
    I'd rather go full chaos monkey and shake out such bugs (/design flaws) than try to avoid them.
    - mchristen 7y ago
      Completely agree there. Some times that luxury doesn't exist though and you just have to get shit done and hope playing fast and loose doesn't come back to bite you.

VectorLock 7y ago

Tuning oomkiller's hitlist makes it much more managable and the new BPF tools to track it make it a ton nicer than a system grinding into swap.

scottlamb 7y ago

> The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. I won't say there's no circumstance in which the OOM killer is worse, but in general I disagree with you. A stateful server application: maybe a standard ACID DBMS like PostgreSQL. I'd much rather it get killed (failing whatever transactions were in progress) and restart in a healthy state than continue to run slowly. Here I'm relying on a correct implementation of Durability, but if you don't have that, you have other problems. Even more so for a quorum-based database server in which the leader dying (and another taking over) is much better than continuing to run slowly. And yes, even more so in stateless servers, particularly when there are several of them. Going down and restarting quickly (a brief loss of capacity, a brief spike of errors if the client doesn't retry) is a lot better than continuing to be slow (a sustained loss of capacity, possibly leading to cascading overload; and user-visible latency problems if there's no hedging). For something like a desktop application, it's debatable, but personally I'd still rather endure the up-front nuisance of having to restart it than have the OS try to paper over it until I finally realize why it's so painful and do something about it. (Insert anecdote about frog boiling here; I hear frogs actually do jump out before they boil, but yet somehow the story still rings true, if you know what I mean.)

mchristen 7y ago
Databases are just one kind of stateful server application though, and yes I would agree with you for that usecase relying on it's underlying durability is a good thing, because it's built with that in mind. Instead I would like to point out the ubiquity of things like nodejs, which make it all to easy for under-experienced developers to shoot themselves in the foot w/ a chaingun with respect to data design and management. It's not really the internal state of the nodejs process itself, it's the external state that it is inevitably being manipulated in the node process.
- scottlamb 7y ago
  I'd rather go full chaos monkey and shake out such bugs (/design flaws) than try to avoid them.
  - mchristen 7y ago
    Completely agree there. Some times that luxury doesn't exist though and you just have to get shit done and hope playing fast and loose doesn't come back to bite you.