5 ms·

Yeah. I didn't really think of it as a bug at first, but I'm glad someone called it that. I wish the system would just kill the browser or low priority processe

by jolmg 7y ago

Yeah. I didn't really think of it as a bug at first, but I'm glad someone called it that. I wish the system would just kill the browser or low priority processes instead of freezing everything in an instant.

magicalhippo 7y ago
Why does it have to kill the browser? Why can't it tell it "nope, no more memory for you" before it's all gone?
- viraptor 7y ago
  It can. System-wide it's one extreme or another. With overcommit enabled, you'll pretty much never get refused. With overcommit disabled, you'll get refusals was soon as you reach the max memory, which means lots of mapped pages wasting space. The middle ground is your own config unfortunately - cgroups can limit available memory, but you'll have to set it up by hand.
  - Annatar 7y ago
    An operating system should never overallocate memory because one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application.
    - angelsl 7y ago
      Then you disable overcommit. For the general case though, because of the way programs have been written, it is easier to have overcommit on.
    - rcxdude 7y ago
      Windows will never overcommit memory. As a result on my PC unless I dedicate a substantial (over 20%) fraction of my drive to swap (most of which will never, ever be touched) I will 'run out of RAM' far before even half of the physical RAM in my PC has been used. This seems extremely wasteful.
    - viraptor 7y ago
      Mapped memory is part of that lying. If you want to use efficient mmap of huge files without lots of manual bookkeeping, you don't want the system to actually allocate as much memory as you specify. With overcommit off you're setting aside memory you're almost never going to use. Many databases rely on the overcommit being possible.
      - garethrowlands 7y ago
        Isn't the memory for memory mapped files backed by the files themselves? Certainly they are on Windows.
        viraptor 7y ago
        Partially. Without overcommit the system needs to guarantee you have enough backing memory to allow you changing the whole file before the drive writes back anything.
    - zzzcpan 7y ago
      > one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application. You cannot make reliable systems thinking like that. The point of resilience is __not__ to rely on correctness of kernel's advertised behavior nor correctness of your assumptions about it.
      - Annatar 7y ago
        Actually I can and have system engineered ultra-reliable production systems which ran over ten years without issues: I'd come back to work at a former employer and the system would still be there, serving. This was not an isolated scenario. One cannot build highly available systems and networks on unreliable, lying software. Basic guarantees are required. If one has 10,000 systems and one loses power on even a portion of such lying systems without basic guarantees, no amount of distribution will guarantee data consistency. I'm no stranger to building very large distributed clusters, but those builds start with an operating system and software which do not overcommit and which are paranoid about data integrity and correctness of operation. In fact, I'm specialized in designing such networks and systems, from hardware to storage all the way up to application software.
- jerkstate 7y ago
  There's the rub. Different people in different situations will want different outcomes from a low-memory situation. One of those different outcomes has to be the default.
  - jolmg 7y ago
    Really, I'd be fine with anything dying as long as it does not freeze. As it stands, when this happens, the only option left is to force a poweroff.
  - Annatar 7y ago
    Actually most Linux users are blissfully unaware that Linux overallocates memory and of those that know about it, very few understand the consequences to applications (such as data corruption or data loss). I doubt anyone who fully understood the consequences would want to run their infrastructure on something so unreliable and I doubt it because it's logical to me that they wouldn't.
    - zzzcpan 7y ago
      You got it backwards. Those of us who care about reliability would never rely on reliability of a single system to safely store data. And overallocating memory is not even remotely that important compared to everything else that can go wrong on a system.
      - Annatar 7y ago
        One cannot build a reliable system on unreliable software. If you believe you can, there is a lesson in store waiting for you.
        jolmg 7y ago
        IPv4 on its own is not reliable, but TCP adds reliability on top of it. Aren't distributed systems in general also designed to be reliable despite being composed of unreliable components?
- JoshTriplett 7y ago
  You can configure Linux to return ENOMEM errors. The theory is that most applications will effectively treat this as fatal and die; so, would you rather kill the app that happened to make the most recent memory allocation or would you rather kill the app using all the memory (and have some control over keeping processes like sshd from dying)? I've tried using a VM with overcommit turned off and a modest amount of memory. Among other things, my mail reader, mutt, used more than half the system's memory when looking at my mail archive, so it couldn't fork to exec an editor to write a new mail.
  - magicalhippo 7y ago
    Why can't it be smart about it? Processes have CPU priorities, why can't they have memory allocation priorities? It doesn't have to wait until it actually runs out before issuing ENOMEM. Most processes don't deserve to starve the system completely.
    - gfv 7y ago
      The fundamental problem here is that the kernel can't time-travel into the future far enough to make sure no higher-priority processes will try to allocate more memory, as the system has to make the decision to return ENOMEM at the time of the allocation. There is a reasonable(-ish?) default to deny allocations that request more memory than currently available (using the Linux definition of available, that is total memory without the sum of resident sets of all processes and a bit more). Of course, this limit only works properly if other processes do not add anything to their working sets, but again, the kernel's fortune-telling abilities are quite limited.
      - magicalhippo 7y ago
        It can't look into the future, but it certainly could use the information from the past. Has this application already allocated 90% of the memory? Has it been steadily growing the allocation without releasing much back? Well then, why let the system run out completely? Why not stop it at say 10% or 5% left?
- 8bitsrule 7y ago
  IMO, that's exactly what it should do. The time when RAM was expensive is gone. Hard-drive swap was always more than a bit of a kludge. So the OS needs to list available options to the user (they -should- choose), and tell apps 'you're SOL, so do whatever you need to do."

by jolmg 7y ago

magicalhippo 7y ago

Why does it have to kill the browser? Why can't it tell it "nope, no more memory for you" before it's all gone?

viraptor 7y ago
It can. System-wide it's one extreme or another. With overcommit enabled, you'll pretty much never get refused. With overcommit disabled, you'll get refusals was soon as you reach the max memory, which means lots of mapped pages wasting space. The middle ground is your own config unfortunately - cgroups can limit available memory, but you'll have to set it up by hand.
- Annatar 7y ago
  An operating system should never overallocate memory because one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application.
  - angelsl 7y ago
    Then you disable overcommit. For the general case though, because of the way programs have been written, it is easier to have overcommit on.
  - rcxdude 7y ago
    Windows will never overcommit memory. As a result on my PC unless I dedicate a substantial (over 20%) fraction of my drive to swap (most of which will never, ever be touched) I will 'run out of RAM' far before even half of the physical RAM in my PC has been used. This seems extremely wasteful.
  - viraptor 7y ago
    Mapped memory is part of that lying. If you want to use efficient mmap of huge files without lots of manual bookkeeping, you don't want the system to actually allocate as much memory as you specify. With overcommit off you're setting aside memory you're almost never going to use. Many databases rely on the overcommit being possible.
    - garethrowlands 7y ago
      Isn't the memory for memory mapped files backed by the files themselves? Certainly they are on Windows.
      - viraptor 7y ago
        Partially. Without overcommit the system needs to guarantee you have enough backing memory to allow you changing the whole file before the drive writes back anything.
  - zzzcpan 7y ago
    > one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application. You cannot make reliable systems thinking like that. The point of resilience is __not__ to rely on correctness of kernel's advertised behavior nor correctness of your assumptions about it.
    - Annatar 7y ago
      Actually I can and have system engineered ultra-reliable production systems which ran over ten years without issues: I'd come back to work at a former employer and the system would still be there, serving. This was not an isolated scenario. One cannot build highly available systems and networks on unreliable, lying software. Basic guarantees are required. If one has 10,000 systems and one loses power on even a portion of such lying systems without basic guarantees, no amount of distribution will guarantee data consistency. I'm no stranger to building very large distributed clusters, but those builds start with an operating system and software which do not overcommit and which are paranoid about data integrity and correctness of operation. In fact, I'm specialized in designing such networks and systems, from hardware to storage all the way up to application software.
jerkstate 7y ago
There's the rub. Different people in different situations will want different outcomes from a low-memory situation. One of those different outcomes has to be the default.
- jolmg 7y ago
  Really, I'd be fine with anything dying as long as it does not freeze. As it stands, when this happens, the only option left is to force a poweroff.
- Annatar 7y ago
  Actually most Linux users are blissfully unaware that Linux overallocates memory and of those that know about it, very few understand the consequences to applications (such as data corruption or data loss). I doubt anyone who fully understood the consequences would want to run their infrastructure on something so unreliable and I doubt it because it's logical to me that they wouldn't.
  - zzzcpan 7y ago
    You got it backwards. Those of us who care about reliability would never rely on reliability of a single system to safely store data. And overallocating memory is not even remotely that important compared to everything else that can go wrong on a system.
    - Annatar 7y ago
      One cannot build a reliable system on unreliable software. If you believe you can, there is a lesson in store waiting for you.
      - jolmg 7y ago
        IPv4 on its own is not reliable, but TCP adds reliability on top of it. Aren't distributed systems in general also designed to be reliable despite being composed of unreliable components?
JoshTriplett 7y ago
You can configure Linux to return ENOMEM errors. The theory is that most applications will effectively treat this as fatal and die; so, would you rather kill the app that happened to make the most recent memory allocation or would you rather kill the app using all the memory (and have some control over keeping processes like sshd from dying)? I've tried using a VM with overcommit turned off and a modest amount of memory. Among other things, my mail reader, mutt, used more than half the system's memory when looking at my mail archive, so it couldn't fork to exec an editor to write a new mail.
- magicalhippo 7y ago
  Why can't it be smart about it? Processes have CPU priorities, why can't they have memory allocation priorities? It doesn't have to wait until it actually runs out before issuing ENOMEM. Most processes don't deserve to starve the system completely.
  - gfv 7y ago
    The fundamental problem here is that the kernel can't time-travel into the future far enough to make sure no higher-priority processes will try to allocate more memory, as the system has to make the decision to return ENOMEM at the time of the allocation. There is a reasonable(-ish?) default to deny allocations that request more memory than currently available (using the Linux definition of available, that is total memory without the sum of resident sets of all processes and a bit more). Of course, this limit only works properly if other processes do not add anything to their working sets, but again, the kernel's fortune-telling abilities are quite limited.
    - magicalhippo 7y ago
      It can't look into the future, but it certainly could use the information from the past. Has this application already allocated 90% of the memory? Has it been steadily growing the allocation without releasing much back? Well then, why let the system run out completely? Why not stop it at say 10% or 5% left?
8bitsrule 7y ago
IMO, that's exactly what it should do. The time when RAM was expensive is gone. Hard-drive swap was always more than a bit of a kludge. So the OS needs to list available options to the user (they -should- choose), and tell apps 'you're SOL, so do whatever you need to do."