4 ms·

An operating system should never overallocate memory because one cannot build reliable applications and infrastructure on top of a kernel which is lying to the

An operating system should never overallocate memory because one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application.

angelsl 7y ago
Then you disable overcommit. For the general case though, because of the way programs have been written, it is easier to have overcommit on.
rcxdude 7y ago
Windows will never overcommit memory. As a result on my PC unless I dedicate a substantial (over 20%) fraction of my drive to swap (most of which will never, ever be touched) I will 'run out of RAM' far before even half of the physical RAM in my PC has been used. This seems extremely wasteful.
viraptor 7y ago
Mapped memory is part of that lying. If you want to use efficient mmap of huge files without lots of manual bookkeeping, you don't want the system to actually allocate as much memory as you specify. With overcommit off you're setting aside memory you're almost never going to use. Many databases rely on the overcommit being possible.
- garethrowlands 7y ago
  Isn't the memory for memory mapped files backed by the files themselves? Certainly they are on Windows.
  - viraptor 7y ago
    Partially. Without overcommit the system needs to guarantee you have enough backing memory to allow you changing the whole file before the drive writes back anything.
zzzcpan 7y ago
> one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application. You cannot make reliable systems thinking like that. The point of resilience is __not__ to rely on correctness of kernel's advertised behavior nor correctness of your assumptions about it.
- Annatar 7y ago
  Actually I can and have system engineered ultra-reliable production systems which ran over ten years without issues: I'd come back to work at a former employer and the system would still be there, serving. This was not an isolated scenario. One cannot build highly available systems and networks on unreliable, lying software. Basic guarantees are required. If one has 10,000 systems and one loses power on even a portion of such lying systems without basic guarantees, no amount of distribution will guarantee data consistency. I'm no stranger to building very large distributed clusters, but those builds start with an operating system and software which do not overcommit and which are paranoid about data integrity and correctness of operation. In fact, I'm specialized in designing such networks and systems, from hardware to storage all the way up to application software.

An operating system should never overallocate memory because one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application.

angelsl 7y ago

Then you disable overcommit. For the general case though, because of the way programs have been written, it is easier to have overcommit on.

rcxdude 7y ago

Windows will never overcommit memory. As a result on my PC unless I dedicate a substantial (over 20%) fraction of my drive to swap (most of which will never, ever be touched) I will 'run out of RAM' far before even half of the physical RAM in my PC has been used. This seems extremely wasteful.

viraptor 7y ago

Mapped memory is part of that lying. If you want to use efficient mmap of huge files without lots of manual bookkeeping, you don't want the system to actually allocate as much memory as you specify. With overcommit off you're setting aside memory you're almost never going to use. Many databases rely on the overcommit being possible.

garethrowlands 7y ago
Isn't the memory for memory mapped files backed by the files themselves? Certainly they are on Windows.
- viraptor 7y ago
  Partially. Without overcommit the system needs to guarantee you have enough backing memory to allow you changing the whole file before the drive writes back anything.

zzzcpan 7y ago

> one cannot build reliable applications and infrastructure on top of a kernel which is lying to the application. You cannot make reliable systems thinking like that. The point of resilience is __not__ to rely on correctness of kernel's advertised behavior nor correctness of your assumptions about it.

Annatar 7y ago
Actually I can and have system engineered ultra-reliable production systems which ran over ten years without issues: I'd come back to work at a former employer and the system would still be there, serving. This was not an isolated scenario. One cannot build highly available systems and networks on unreliable, lying software. Basic guarantees are required. If one has 10,000 systems and one loses power on even a portion of such lying systems without basic guarantees, no amount of distribution will guarantee data consistency. I'm no stranger to building very large distributed clusters, but those builds start with an operating system and software which do not overcommit and which are paranoid about data integrity and correctness of operation. In fact, I'm specialized in designing such networks and systems, from hardware to storage all the way up to application software.