4 ms·
It's still done. I did the code review for the guy who wrote it for our server. If we're crashing due to an OOM situation we still have enough memory to write o
by melech_ric 4y ago
It's still done. I did the code review for the guy who wrote it for our server. If we're crashing due to an OOM situation we still have enough memory to write out a minidump.
- touisteur 4y agoThe pain here is to test and keep testing these paths. Same as some 'free disk space' monitoring...
- melech_ric 4y agoI agree. However, in our case, that pain has proven to be less than not having those minidumps.
- touisteur 4y agoOh for me too, but I wish it was easier, like something supported seriously by my language of choice, my OS of choice or my libc of choice, with the flick of an flag... I don't retest the whole libc and kernel every time (although, err, someone should, right? :-)
- _vvhw 4y agoJoran from the TigerBeetle team here! TigerBeetle uses Deterministic Simulation Testing to test and keep testing these paths. Fuzzing and static allocation are force multipliers when applied together, because you can now flush out leaks and deadlocks in testing, rather than letting these spillover into production. Without static allocation, it's a little harder to find leaks in testing, because the limits that would define a leak are not explicit.
- polskibus 4y agoCan you provide some pointers regarding the simulation testing for more information?
- _vvhw 4y agoSure! Here's an overview with references to the simulator source, and links to resources from FoundationDB and Dropbox: https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/HACKING.md#simulation-tests https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/... We also have a $20k bounty that you can take part in, where you can run the simulator yourself.
- srcreigh 4y agoWhat language? What does the server do?
- melech_ric 4y agoC++ on Windows. It's an old codebase that has been used to run custom test hardware. We also target Linux with the same codebase, but the environment is a bit different. On Linux we're using a system slice and we're not doing minidumps.
- kotlin2 4y agoWhat environment is the server running in? Can you prevent the Linux OOM killer from killing your process?
- abeyer 4y agoThe OOM killer can be disabled completely on a system, or a process can be excluded from it by setting an OOM flag under `/proc/<pid>` It's not generally considered good practice in production, but I think that's largely because most software does not do what is being suggested here... in the scenario where you have, seems like the right thing to do.
- throwaway09223 4y agoIt's definitely good practice in production and is often necessary. The techniques mentioned above will (perhaps surprisingly) not eliminate errors related to OOM, due to the nature of virtual memory. Your program can OOM at runtime even if you malloc all your resources up front, because allocating address space is not the same thing as allocating memory. In fact, memory can be deallocated (swapped out), and then your application may OOM when it tries to access memory it has previously used successfully. Without looking, I can confidently say that tigerbeetle does in fact dynamically allocate memory -- even if it does not call malloc at runtime.
- _vvhw 4y agoWe're aware of this, in fact, and do have a plan to address virtual memory. To be fair, it's really the kernel being dynamic here, not TigerBeetle.
- throwaway09223 4y agoOh for sure. Not casting any shade cast at all on your work - I am really happy to see what you're doing. This kind of thing has a lot of value even in a virtual memory environment.
- 4y ago