5 ms·

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for succe

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.

alexgartrell 5y ago
Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing your way, but it'd be phenomenally more expensive. I know this because every thing we introduce to the hierarchy in practice makes computing phenomenally less expensive.
- cturner 5y ago
  "We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.
  - pjc50 5y ago
    Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975 https://news.ycombinator.com/item?id=9653975 For ages cygwin had to emulate it manually by hand-copying the process.
  - throwawaylinux 5y ago
    Linux can prevent overcommit. And that's not the reason for much faster process creation than Windows as far as I know.
  - Rusky 5y ago
    I'm not sure there's any causality between Windows preventing overcommit and Windows having slow process creation- there's nothing inherently slower about CreateProcess()/posix_spawn() than fork() + exec() as an API. It seems more that overcommit is a workaround for the way fork() can potentially lead to copying the entire address space into new pages, but usually doesn't. Because CreateProcess() knows more precisely how much to allocate before it returns to the child process, it can just reserve that amount and signal an error immediately if there's not enough memory+swap to back it. (And on the other hand, Windows has a lot of legacy and backwards compatibility behavior around processes that could easily explain the slower process creation independent of the API.)
joosters 5y ago
Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.
- giomasce 5y ago
  I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.
  - gpderetta 5y ago
    They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.
    - OldHand2018 5y ago
      This paper [1] argues that fork needs to be deprecated in favor of posix_spawn() or other more modern solutions. It claims that, among many other reasons, that fork encourages overcommit and that programs such as Redis are extraordinarily constrained if overcommit is disabled - because of fork. [1] https://dl.acm.org/doi/10.1145/3317550.3321435 https://dl.acm.org/doi/10.1145/3317550.3321435
    - scottlamb 5y ago
      > Vfork is usually a better solution. I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions." So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In practice, I believe it's fine, and on glibc posix_spawn is apparently written in terms of vfork, but libc is allowed to make assumptions that portable programs shouldn't. Story time: I once worked on a Unix-based system with no MMU. The fork implementation did something insane: it looked for things that were possibly pointers (four-byte aligned memory locations which can be interpreted as valid physical memory locations on this system allocated to that program) and adjusted them. It was kind of like a conservative GC in that it treated anything that looked like a pointer as if it were a pointer. But no reasonable person would call modifying memory that may or may not be pointers to be "conservative". Amazingly, it worked most of the time, but sometimes strings got corrupted, and as a workaround there were a bunch of places where string buffers were fully zeroed where just a NUL byte would otherwise do. This was early in my career, and I didn't design the system anyway. If I were working on it today I'd remove the fork implementation and make everything use vfork and/or posix_spawn instead. Apparently that's what posix_spawn was made for. Another weird problem with the lack of MMU: the compiler also didn't use register-based addressing (what's the term? like position-independent code but for the data segment?), so global variables were truly global, not just global to the process. A bunch of code needed to be "deglobalized" to deal with this. I wanted to improve the compiler, but long story short I got offered another job first.
OldHand2018 5y ago
Let's say you malloc some memory and the computer actually has everything available. Everything is great, up until some other process on your system does a fork bomb of an infinitely recursive program that allocates nothing on the heap. You've just got a whole lot of quickly growing stacks hoovering up your physical memory pages.
- tsimionescu 5y ago
  Stack vs Heap is a userspace level concept. As far as the kernel memory manager knows, it's all just allocated memory - some is allocated by the process- or thread-spawning routines, some is allocated by malloc(), but they're using the exact same pool. If overcommit is disabled and someone has allocated most system memory, fork() and exec() and pthread_create() etc. will theoretically fail with ENOMEM. A bigger problem on Linux at least is that the kernel will swap out all possible memory before returning an allocation error. Even if you have not allocated any swap space, it will swap out any memory mapped files. And even if none of your programs have explicitly mmap()ed anything, the actual application code is mapped, so it will start swapping out all code pages to disk before it refuses an allocation. And now this means that your system has become entirely unusable and will have to be hard rebooted, because it is now swapping data to and from disk on every instruction execution after every context switch. At least your CPU will get to stay nice and cool for a while.
  - Dylan16807 5y ago
    It's really unfortunate that you can't set minimum and maximum disk cache on Linux.

alexgartrell 5y ago

Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing your way, but it'd be phenomenally more expensive. I know this because every thing we introduce to the hierarchy in practice makes computing phenomenally less expensive.

cturner 5y ago
"We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.
- pjc50 5y ago
  Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975 https://news.ycombinator.com/item?id=9653975 For ages cygwin had to emulate it manually by hand-copying the process.
- throwawaylinux 5y ago
  Linux can prevent overcommit. And that's not the reason for much faster process creation than Windows as far as I know.
- Rusky 5y ago
  I'm not sure there's any causality between Windows preventing overcommit and Windows having slow process creation- there's nothing inherently slower about CreateProcess()/posix_spawn() than fork() + exec() as an API. It seems more that overcommit is a workaround for the way fork() can potentially lead to copying the entire address space into new pages, but usually doesn't. Because CreateProcess() knows more precisely how much to allocate before it returns to the child process, it can just reserve that amount and signal an error immediately if there's not enough memory+swap to back it. (And on the other hand, Windows has a lot of legacy and backwards compatibility behavior around processes that could easily explain the slower process creation independent of the API.)

joosters 5y ago

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

giomasce 5y ago
I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.
- gpderetta 5y ago
  They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.
  - OldHand2018 5y ago
    This paper [1] argues that fork needs to be deprecated in favor of posix_spawn() or other more modern solutions. It claims that, among many other reasons, that fork encourages overcommit and that programs such as Redis are extraordinarily constrained if overcommit is disabled - because of fork. [1] https://dl.acm.org/doi/10.1145/3317550.3321435 https://dl.acm.org/doi/10.1145/3317550.3321435
  - scottlamb 5y ago
    > Vfork is usually a better solution. I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions." So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In practice, I believe it's fine, and on glibc posix_spawn is apparently written in terms of vfork, but libc is allowed to make assumptions that portable programs shouldn't. Story time: I once worked on a Unix-based system with no MMU. The fork implementation did something insane: it looked for things that were possibly pointers (four-byte aligned memory locations which can be interpreted as valid physical memory locations on this system allocated to that program) and adjusted them. It was kind of like a conservative GC in that it treated anything that looked like a pointer as if it were a pointer. But no reasonable person would call modifying memory that may or may not be pointers to be "conservative". Amazingly, it worked most of the time, but sometimes strings got corrupted, and as a workaround there were a bunch of places where string buffers were fully zeroed where just a NUL byte would otherwise do. This was early in my career, and I didn't design the system anyway. If I were working on it today I'd remove the fork implementation and make everything use vfork and/or posix_spawn instead. Apparently that's what posix_spawn was made for. Another weird problem with the lack of MMU: the compiler also didn't use register-based addressing (what's the term? like position-independent code but for the data segment?), so global variables were truly global, not just global to the process. A bunch of code needed to be "deglobalized" to deal with this. I wanted to improve the compiler, but long story short I got offered another job first.

OldHand2018 5y ago

Let's say you malloc some memory and the computer actually has everything available. Everything is great, up until some other process on your system does a fork bomb of an infinitely recursive program that allocates nothing on the heap. You've just got a whole lot of quickly growing stacks hoovering up your physical memory pages.

tsimionescu 5y ago
Stack vs Heap is a userspace level concept. As far as the kernel memory manager knows, it's all just allocated memory - some is allocated by the process- or thread-spawning routines, some is allocated by malloc(), but they're using the exact same pool. If overcommit is disabled and someone has allocated most system memory, fork() and exec() and pthread_create() etc. will theoretically fail with ENOMEM. A bigger problem on Linux at least is that the kernel will swap out all possible memory before returning an allocation error. Even if you have not allocated any swap space, it will swap out any memory mapped files. And even if none of your programs have explicitly mmap()ed anything, the actual application code is mapped, so it will start swapping out all code pages to disk before it refuses an allocation. And now this means that your system has become entirely unusable and will have to be hard rebooted, because it is now swapping data to and from disk on every instruction execution after every context switch. At least your CPU will get to stay nice and cool for a while.
- Dylan16807 5y ago
  It's really unfortunate that you can't set minimum and maximum disk cache on Linux.