5 ms·

The idea of having programs that don't crash under memory pressure is one of the things that I really hope developers will start striving for more in the near f

The idea of having programs that don't crash under memory pressure is one of the things that I really hope developers will start striving for more in the near future. A database that doesn't OOM crash is great, and the same is also true for more consumer-grade software. I have 32gb of RAM on my PC and whenever I'm encoding a video with DaVinci Resolve, I always see Firefox and Discord crash almost instantly.

colejohnson66 4y ago
OOM is usually a good reason to crash. Sure, a DB might be able to do something creative with its cache, but if you can't allocate, there's not much the average program can do anymore. I guess browsers can kill their child processes, but that's not a long-term solution. If you're out of memory, your computer will be lagging from paging/swapping memory in-and-out, so killing yourself helps relieve some pressure. Also, are you sure it's the programs failing and not some OOM killer coming around? Could your pagefile/swap be full?
- advisedwang 4y ago
  The article says it rejects incoming connections. that seems like a much better behaviour than OOM - clients can back-off and retry and keep the overall system stable.
  - _vvhw 4y ago
    Thanks! Joran from the TigerBeetle team here. This was in fact one of our motivations for static allocation—thinking about how best to handle overload from the network, while remaining stable. The Google SRE book has a great chapter on this called "Handling Overload" and this had an impact on us. We were thinking, well, how do we get this right for a database? We also wanted to make explicit what is often implicit, so that the operator has a clear sense of how to provision their API layer around TigerBeetle.
  - lmm 4y ago
    I can easily imagine that kind of design getting into a state where it's "up" but not accepting any connections indefinitely (if it's using just enough memory to run itself, but doesn't have enough to accept any connections). Crashing early is often a good way to reduce the state space and ensure consistent behaviour, since you will always have to handle crashing anyway.
    - wahern 4y ago
      Crashing and losing all existing connections often merely leads to an avalanche of subsequent requests, accelerating overload of your systems. I've seen this countless times. Being crash tolerant is one thing. But "crash early, crash often" is absolutely horrible advice.
      - lmm 4y ago
        > Crashing and losing all existing connections often merely leads to an avalanche of subsequent requests, accelerating overload of your systems. I've seen this countless times. Sure, but again, that's a scenario that you need to handle anyway. > Being crash tolerant is one thing. But "crash early, crash often" is absolutely horrible advice. I've found it to be good advice. It's a big part of why Erlang systems have been so reliable for decades.
        wahern 4y ago
        In Erlang a "process" is more akin to a thread or fiber in most other languages, and "crashing" more like an exception that is caught just across a logical task boundary. Importantly, in Erlang a process "crashing" doesn't kill all other connections and tasks within the kernel-level Erlang process. And that's why Erlang is so resilient, precisely because the semantics of the language make it easier to isolate tasks and subtasks, minimizing blast radius and the risk of reaching an inconsistent or unrecoverable state. I often use Lua (as a high-level glue language) to accomplish something similar: I can run a task on a coroutine, and if a constraint or assertion throws an error (exceptions in Lua are idiomatically used sparingly, similar to how they're used in Go), it's much easier to recover. This is true even for malloc failures, which can be caught at the same boundaries (pcall, coroutine.resume, etc) as any other error.[1] It's also common to use multiple separate Lua VM states in the same kernel process, communicating using sockets, data-copying channels, etc, achieving isolation behaviors even closer to what Erlang provides, along with the potential performance costs. [1] While more tricky in a language like Lua, as long as your steady-state--i.e. event loop, etc--is free of dynamic allocations, then malloc failure is trivially recoverable. The Lua authors are careful to document which interfaces and operations might allocate, and to keep a core set of operations allocation free. Of course, you still need to design your own components likewise, and ideally such that allocations for connection request, subtasks, etc can be front-loaded (RAII style), or if not then isolated behind convenient recovery points. Erlang makes much of this discipline perfunctory.
- yellowapple 4y ago
  > Could your pagefile/swap be full? This presumes that one is indeed using a pagefile or swap partition. I know of quite a few folks who skip that on SSDs out of fear of SSD wear. Personally, in this day and age I don't really give a damn about SSD wear (that's what backups are for), so I'll happily create a swap partition, but not everyone is as comfortable with that.
- perryizgr8 4y ago
  Software for critical infrastructure often handles no/low memory situations in a graceful manner. I've worked on network switch software where we were expected to work indefinitely with zero remaining memory. A single packet drop is unacceptable, at 10-40G line rate. Any new malloc needs to be justified with a design document. So yeah, average user software can do a lot, it's just that we've given up on reliability and robustness as an industry. That's why sometimes your phone fails to make a 911 call, and sometimes you need to reset your car.
- slaymaker1907 4y ago
  When SQL Server goes OOM, it's often because the system can't free memory quickly enough, not that there is literally not enough memory to continue operating. Sure, it could be more aggressive with freeing memory, but at some point it just makes more sense to restart the entire system so that overall performance doesn't nosedive.
pphysch 4y ago
It would be nice if you could specify e.g. a Postgres table be backed by a (really big) "ring buffer" rather than the usual storage. Maybe that breaks some relational features on the table, but being able to painlessly query the most recent X records for some interesting transient log data would be nice.
- ok_dad 4y ago
  I ended up using a ring buffer in Redis recently to store the last N values of some database lookups that are slow. It works great! Didn't take that much logic, either, since Redis has easy transactions. I just have the "current buffer index 'i'" as a key-value, the "buffer length 'N'" as a key-value, then 'N' numbered entries as key-values. I prefix these with the data identifier that it caches, so you can have several ring buffers active. you can do this with a Postgres table just as easily, I assume! I also am building a ring buffer data structure for my hobby database that I use for personal stuff that uses FoundationDB [though I have been unhappy with the state of the FDB Go bindings, which are somewhat out of date versus the C client library for FDB and aren't very Go-like in design (the C libraries are the "gold standard" and they use FFI in all the other bindings to connect to those)]. Added later: it was a bit harder to deal with resizing the buffer, so when that happens I do some data shifing/dropping depending on which way we're resizing, effectively initializing the buffer anew because that was easier (again, a Redis transaction saves me a lot of concurrency work!), but lucky for me we would only resize each buffer at most 1 time per minute due to how our configs work.
slaymaker1907 4y ago
What would happen in your utopia is that DaVinci Resolve would fail to start because Firefox or Discord would reserve too much memory. Or maybe Firefox would demand you close open tabs before allowing you to open a new one. We have a solution to the problem you describe, it's called paging to disk and hoping that we won't need to read or write to that memory very often (which works great for something like Firefox where a bunch of pages are sitting idly in the background).
- kristoff_it 4y ago
  Allocating upfront is only one technique. Somehow DaVinci Resolve is capable of using all the free memory it can find and not more than that, otherwise it would never be able to complete any rendering job. That's one "utopic" program that's pretty real. > We have a solution to the problem you describe, it's called paging to disk and hoping No. The solution is to be smart about memory usage, and to handle failures when trying to allocate memory. DaVinci would become unusable if it started swapping when rendering. Maybe for some programs swapping is a fine solution, but the argument that software engineers shouldn't concern themselves with designing for OOM conditions is absolutely ridiculous. That's not a 'utopia', that's what we should be concretely be striving for.
  - mananaysiempre 4y ago
    It’s difficult to see how that would be possible without a hard break from legacy systems when fork/exec practically forces overcommit and very dubious resource accounting on us. (One consumer system that handled OOM conditions with remarkable grace was Symbian, but of course it forced an extremely peculiar programming style.)
    - kristoff_it 4y ago
      That's a fair point, but it's also true that we're never going to break away from legacy systems until we start writing software that has a chance of functioning correctly when running out of resources. The good news is that we do have ecosystems where you don't have overcommit, like webassembly and embedded devices, and if you're writing libraries meant to also work on those platforms, then you will have some good building blocks that can be used to iterate towards more robust software. That's how you're expected to approach library writing in Zig.

colejohnson66 4y ago

OOM is usually a good reason to crash. Sure, a DB might be able to do something creative with its cache, but if you can't allocate, there's not much the average program can do anymore. I guess browsers can kill their child processes, but that's not a long-term solution. If you're out of memory, your computer will be lagging from paging/swapping memory in-and-out, so killing yourself helps relieve some pressure. Also, are you sure it's the programs failing and not some OOM killer coming around? Could your pagefile/swap be full?

advisedwang 4y ago
The article says it rejects incoming connections. that seems like a much better behaviour than OOM - clients can back-off and retry and keep the overall system stable.
- _vvhw 4y ago
  Thanks! Joran from the TigerBeetle team here. This was in fact one of our motivations for static allocation—thinking about how best to handle overload from the network, while remaining stable. The Google SRE book has a great chapter on this called "Handling Overload" and this had an impact on us. We were thinking, well, how do we get this right for a database? We also wanted to make explicit what is often implicit, so that the operator has a clear sense of how to provision their API layer around TigerBeetle.
- lmm 4y ago
  I can easily imagine that kind of design getting into a state where it's "up" but not accepting any connections indefinitely (if it's using just enough memory to run itself, but doesn't have enough to accept any connections). Crashing early is often a good way to reduce the state space and ensure consistent behaviour, since you will always have to handle crashing anyway.
  - wahern 4y ago
    Crashing and losing all existing connections often merely leads to an avalanche of subsequent requests, accelerating overload of your systems. I've seen this countless times. Being crash tolerant is one thing. But "crash early, crash often" is absolutely horrible advice.
    - lmm 4y ago
      > Crashing and losing all existing connections often merely leads to an avalanche of subsequent requests, accelerating overload of your systems. I've seen this countless times. Sure, but again, that's a scenario that you need to handle anyway. > Being crash tolerant is one thing. But "crash early, crash often" is absolutely horrible advice. I've found it to be good advice. It's a big part of why Erlang systems have been so reliable for decades.
      - wahern 4y ago
        In Erlang a "process" is more akin to a thread or fiber in most other languages, and "crashing" more like an exception that is caught just across a logical task boundary. Importantly, in Erlang a process "crashing" doesn't kill all other connections and tasks within the kernel-level Erlang process. And that's why Erlang is so resilient, precisely because the semantics of the language make it easier to isolate tasks and subtasks, minimizing blast radius and the risk of reaching an inconsistent or unrecoverable state. I often use Lua (as a high-level glue language) to accomplish something similar: I can run a task on a coroutine, and if a constraint or assertion throws an error (exceptions in Lua are idiomatically used sparingly, similar to how they're used in Go), it's much easier to recover. This is true even for malloc failures, which can be caught at the same boundaries (pcall, coroutine.resume, etc) as any other error.[1] It's also common to use multiple separate Lua VM states in the same kernel process, communicating using sockets, data-copying channels, etc, achieving isolation behaviors even closer to what Erlang provides, along with the potential performance costs. [1] While more tricky in a language like Lua, as long as your steady-state--i.e. event loop, etc--is free of dynamic allocations, then malloc failure is trivially recoverable. The Lua authors are careful to document which interfaces and operations might allocate, and to keep a core set of operations allocation free. Of course, you still need to design your own components likewise, and ideally such that allocations for connection request, subtasks, etc can be front-loaded (RAII style), or if not then isolated behind convenient recovery points. Erlang makes much of this discipline perfunctory.
yellowapple 4y ago
> Could your pagefile/swap be full? This presumes that one is indeed using a pagefile or swap partition. I know of quite a few folks who skip that on SSDs out of fear of SSD wear. Personally, in this day and age I don't really give a damn about SSD wear (that's what backups are for), so I'll happily create a swap partition, but not everyone is as comfortable with that.
perryizgr8 4y ago
Software for critical infrastructure often handles no/low memory situations in a graceful manner. I've worked on network switch software where we were expected to work indefinitely with zero remaining memory. A single packet drop is unacceptable, at 10-40G line rate. Any new malloc needs to be justified with a design document. So yeah, average user software can do a lot, it's just that we've given up on reliability and robustness as an industry. That's why sometimes your phone fails to make a 911 call, and sometimes you need to reset your car.
slaymaker1907 4y ago
When SQL Server goes OOM, it's often because the system can't free memory quickly enough, not that there is literally not enough memory to continue operating. Sure, it could be more aggressive with freeing memory, but at some point it just makes more sense to restart the entire system so that overall performance doesn't nosedive.

pphysch 4y ago

It would be nice if you could specify e.g. a Postgres table be backed by a (really big) "ring buffer" rather than the usual storage. Maybe that breaks some relational features on the table, but being able to painlessly query the most recent X records for some interesting transient log data would be nice.

ok_dad 4y ago
I ended up using a ring buffer in Redis recently to store the last N values of some database lookups that are slow. It works great! Didn't take that much logic, either, since Redis has easy transactions. I just have the "current buffer index 'i'" as a key-value, the "buffer length 'N'" as a key-value, then 'N' numbered entries as key-values. I prefix these with the data identifier that it caches, so you can have several ring buffers active. you can do this with a Postgres table just as easily, I assume! I also am building a ring buffer data structure for my hobby database that I use for personal stuff that uses FoundationDB [though I have been unhappy with the state of the FDB Go bindings, which are somewhat out of date versus the C client library for FDB and aren't very Go-like in design (the C libraries are the "gold standard" and they use FFI in all the other bindings to connect to those)]. Added later: it was a bit harder to deal with resizing the buffer, so when that happens I do some data shifing/dropping depending on which way we're resizing, effectively initializing the buffer anew because that was easier (again, a Redis transaction saves me a lot of concurrency work!), but lucky for me we would only resize each buffer at most 1 time per minute due to how our configs work.

slaymaker1907 4y ago

What would happen in your utopia is that DaVinci Resolve would fail to start because Firefox or Discord would reserve too much memory. Or maybe Firefox would demand you close open tabs before allowing you to open a new one. We have a solution to the problem you describe, it's called paging to disk and hoping that we won't need to read or write to that memory very often (which works great for something like Firefox where a bunch of pages are sitting idly in the background).

kristoff_it 4y ago
Allocating upfront is only one technique. Somehow DaVinci Resolve is capable of using all the free memory it can find and not more than that, otherwise it would never be able to complete any rendering job. That's one "utopic" program that's pretty real. > We have a solution to the problem you describe, it's called paging to disk and hoping No. The solution is to be smart about memory usage, and to handle failures when trying to allocate memory. DaVinci would become unusable if it started swapping when rendering. Maybe for some programs swapping is a fine solution, but the argument that software engineers shouldn't concern themselves with designing for OOM conditions is absolutely ridiculous. That's not a 'utopia', that's what we should be concretely be striving for.
- mananaysiempre 4y ago
  It’s difficult to see how that would be possible without a hard break from legacy systems when fork/exec practically forces overcommit and very dubious resource accounting on us. (One consumer system that handled OOM conditions with remarkable grace was Symbian, but of course it forced an extremely peculiar programming style.)
  - kristoff_it 4y ago
    That's a fair point, but it's also true that we're never going to break away from legacy systems until we start writing software that has a chance of functioning correctly when running out of resources. The good news is that we do have ecosystems where you don't have overcommit, like webassembly and embedded devices, and if you're writing libraries meant to also work on those platforms, then you will have some good building blocks that can be used to iterate towards more robust software. That's how you're expected to approach library writing in Zig.