5 ms·

Allocating upfront is only one technique. Somehow DaVinci Resolve is capable of using all the free memory it can find and not more than that, otherwise it woul

Allocating upfront is only one technique. Somehow DaVinci Resolve is capable of using all the free memory it can find and not more than that, otherwise it would never be able to complete any rendering job. That's one "utopic" program that's pretty real. > We have a solution to the problem you describe, it's called paging to disk and hoping No. The solution is to be smart about memory usage, and to handle failures when trying to allocate memory. DaVinci would become unusable if it started swapping when rendering. Maybe for some programs swapping is a fine solution, but the argument that software engineers shouldn't concern themselves with designing for OOM conditions is absolutely ridiculous. That's not a 'utopia', that's what we should be concretely be striving for.

mananaysiempre 4y ago
It’s difficult to see how that would be possible without a hard break from legacy systems when fork/exec practically forces overcommit and very dubious resource accounting on us. (One consumer system that handled OOM conditions with remarkable grace was Symbian, but of course it forced an extremely peculiar programming style.)
- kristoff_it 4y ago
  That's a fair point, but it's also true that we're never going to break away from legacy systems until we start writing software that has a chance of functioning correctly when running out of resources. The good news is that we do have ecosystems where you don't have overcommit, like webassembly and embedded devices, and if you're writing libraries meant to also work on those platforms, then you will have some good building blocks that can be used to iterate towards more robust software. That's how you're expected to approach library writing in Zig.
  - deleted 4y ago
    [deleted]
astrange 4y ago
The purpose of abstractions is so you don't have to think about what's under them. malloc() can fail if you try to allocate many GBs at once, but if you let it fail for 32 byte allocations then what are you even going to do about that? The only solution is to crash your process, but that will lose user data if you haven't saved it yet, and purity isn't worth that.
- kristoff_it 4y ago
  As far as I'm aware, witing to a file doesn't require any memory allocation on any of the major OSs, so you can most definitely save the user's work when running out of memory. > The only solution is to crash your process I disagree. I think this is a bad mindset that we allowed ourselves to slid into. There's a lot more than can be done when an allocation fails, starting from what I mentioned above. That said, I'll agree that at least you need a programming language that can help you design and maintain allocation-free codepaths in your program, and that 's hard if your language has builtins that allocate implicitly, as you will have to avoid them all. Zig has the philosophy of keeping all memory allocations explicit precisely for this reason. In the beginning people used to say that it was too extreme, but now Rust is also embracing this approach in order to be used in the Linux Kernel (one place where the "only solution is to crash" learned helplessness will not be well received), so hopefully there will be even more options in the future.
  - astrange 4y ago
    > As far as I'm aware, witing to a file doesn't require any memory allocation on any of the major OSs, so you can most definitely save the user's work when running out of memory. Course it does. You can’t run any code in a typical app environment without allocating heap objects as you go, and even if you didn’t, anything that accesses a swapped out page is “allocating physical memory”. And once it gets into the kernel, file systems have to allocate their own objects to track new writes. Though if the kernel is failing to allocate memory you have worse problems.
    - kristoff_it 4y ago

mananaysiempre 4y ago

It’s difficult to see how that would be possible without a hard break from legacy systems when fork/exec practically forces overcommit and very dubious resource accounting on us. (One consumer system that handled OOM conditions with remarkable grace was Symbian, but of course it forced an extremely peculiar programming style.)

kristoff_it 4y ago
That's a fair point, but it's also true that we're never going to break away from legacy systems until we start writing software that has a chance of functioning correctly when running out of resources. The good news is that we do have ecosystems where you don't have overcommit, like webassembly and embedded devices, and if you're writing libraries meant to also work on those platforms, then you will have some good building blocks that can be used to iterate towards more robust software. That's how you're expected to approach library writing in Zig.
- deleted 4y ago
  [deleted]

astrange 4y ago

The purpose of abstractions is so you don't have to think about what's under them. malloc() can fail if you try to allocate many GBs at once, but if you let it fail for 32 byte allocations then what are you even going to do about that? The only solution is to crash your process, but that will lose user data if you haven't saved it yet, and purity isn't worth that.

kristoff_it 4y ago
As far as I'm aware, witing to a file doesn't require any memory allocation on any of the major OSs, so you can most definitely save the user's work when running out of memory. > The only solution is to crash your process I disagree. I think this is a bad mindset that we allowed ourselves to slid into. There's a lot more than can be done when an allocation fails, starting from what I mentioned above. That said, I'll agree that at least you need a programming language that can help you design and maintain allocation-free codepaths in your program, and that 's hard if your language has builtins that allocate implicitly, as you will have to avoid them all. Zig has the philosophy of keeping all memory allocations explicit precisely for this reason. In the beginning people used to say that it was too extreme, but now Rust is also embracing this approach in order to be used in the Linux Kernel (one place where the "only solution is to crash" learned helplessness will not be well received), so hopefully there will be even more options in the future.
- astrange 4y ago
  > As far as I'm aware, witing to a file doesn't require any memory allocation on any of the major OSs, so you can most definitely save the user's work when running out of memory. Course it does. You can’t run any code in a typical app environment without allocating heap objects as you go, and even if you didn’t, anything that accesses a swapped out page is “allocating physical memory”. And once it gets into the kernel, file systems have to allocate their own objects to track new writes. Though if the kernel is failing to allocate memory you have worse problems.
  - kristoff_it 4y ago