3 ms·
Got it. To be honest, what I really wanted here is to force the commenter on agreeing that whatever the architecture/framework/etc., there is no magical soluti
by Galanwe 7y ago
Got it.
To be honest, what I really wanted here is to force the commenter on agreeing
that whatever the architecture/framework/etc., there is no magical solution,
and saying that it will "be compensated-for at a higher level" just hides the
only 2 real possibilities:
- Kill the process and restart it somewhere else (no swap)
- Wait for the process to finish anyway (using swap)
Both choices have their own pros and cons, as discussed in other comments. But
arguments such as "it's better to have predictable process termination", and
"under tight latency requirements then paging can push you over that boundary"
are exposed as much less meaningful once the problem is stated in simple terms.
If the process indeed uses more memory than planned, then you have no guarantee
than restarting it will work this time. Worst case, you can even have to
allocate urgently a new node to handle that 1MB of memory above what was
available. Not sure that is really the best solution "under tight latency
requirements".
My experience would indeed be the reverse: if you are faced with unexpected memory
usage growth, then having some swap on the side to allow your process to finish makes
for a much smoother production system than having to restart it all from scratch
somewhere else/at an other time.
In my experience, production is not the place where you want to enforce ideological
concepts such as "a process shall not use more memory than what was planned".
Production is the place where you make compromises and account for everything that
slipped through these concepts, because you have no choice: it _has_ to run.
Staging would be a good place for this though, as it's much less critical if
something fails, and you have more time to fix and account for it.
I really don't want to debug/profile/release/deploy a program in a hurry just because
somehow it ended up using more memory than planned. (or convince infrastructure to
deploy a new node type ASAP to handle that new memory constraint).
Now one could argue that such unplanned memory consumption should have been caught
earlier in the development process, to which I would reply:
- Not everyone is able to properly test programs in a production like environment,
especially when dealing with resource-hungry processes. In our case for instance
a typical staging machine would be a downscaled version of the production one (less
cores, less memory, less bandwidth).
- I tend to plan _also_ for what should _not_ happen (because it will happen). Even
with a perfect environment to calibrate programs before a release, there will be
releases in which the memory usage was not accounted for. Guaranteed.
- russler23 7y agoI’m guessing this is a disagreement on the meaning of “reliable,” and there is room to disagree on that. From my perspective, if your failure model is “it has to run” then you don’t really have a failure model and are just hoping for the best. If you have the resources to improve on that approach and formalize expected failure, then OOM logs are another tool that can give you metrics from production while failing operationally. The real benefit is ease of debugging.