6 ms·

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gi

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

giomasce 5y ago
I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.
- gpderetta 5y ago
  They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.
  - OldHand2018 5y ago
    This paper [1] argues that fork needs to be deprecated in favor of posix_spawn() or other more modern solutions. It claims that, among many other reasons, that fork encourages overcommit and that programs such as Redis are extraordinarily constrained if overcommit is disabled - because of fork. [1] https://dl.acm.org/doi/10.1145/3317550.3321435 https://dl.acm.org/doi/10.1145/3317550.3321435
  - scottlamb 5y ago
    > Vfork is usually a better solution. I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions." So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In practice, I believe it's fine, and on glibc posix_spawn is apparently written in terms of vfork, but libc is allowed to make assumptions that portable programs shouldn't. Story time: I once worked on a Unix-based system with no MMU. The fork implementation did something insane: it looked for things that were possibly pointers (four-byte aligned memory locations which can be interpreted as valid physical memory locations on this system allocated to that program) and adjusted them. It was kind of like a conservative GC in that it treated anything that looked like a pointer as if it were a pointer. But no reasonable person would call modifying memory that may or may not be pointers to be "conservative". Amazingly, it worked most of the time, but sometimes strings got corrupted, and as a workaround there were a bunch of places where string buffers were fully zeroed where just a NUL byte would otherwise do. This was early in my career, and I didn't design the system anyway. If I were working on it today I'd remove the fork implementation and make everything use vfork and/or posix_spawn instead. Apparently that's what posix_spawn was made for. Another weird problem with the lack of MMU: the compiler also didn't use register-based addressing (what's the term? like position-independent code but for the data segment?), so global variables were truly global, not just global to the process. A bunch of code needed to be "deglobalized" to deal with this. I wanted to improve the compiler, but long story short I got offered another job first.
    - pcwalton 5y ago
      No-MMU Linux usually just disables fork and uses vfork instead. The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI).
      - scottlamb 5y ago
        > The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI). Interesting, thanks. That didn't come up on our system—not only did we not use shared libraries but we also linked the whole system into one binary, kernel and all. Link times were atrocious in combination with identical code folding and big VLIW sentences, but it worked.
  - jart 5y ago
    > They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. In other words, you write your C code inside a C string instead of in C.
- joosters 5y ago
  It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on… the flexibility of a separate fork() then exec() means you can set up the initial state of a new process exactly as you want, by doing whatever work is needed between the two calls. If you merge them into one, then you will never be able to encapsulate all of that.
  - saagarjha 5y ago
    I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)
    - marcan_42 5y ago
      Are there any things that are illegal between fork and exec? It is perfectly legal to exec whenever you want, and it it perfectly legal to fork whenever you want. I'm not aware of any requirements around the fork/exec sequence.
      - tsimionescu 5y ago
        For a single-threaded process, sure. But in a multi threaded context, almost any operation done in the child after fork() can royally mess up the system. There are many versions of libcs where malloc() after fork() is likely to deadlock between the parent and child processes, as they share the internal malloc() locks.
        marcan_42 5y ago
        Ah, yes, multithreading. That always interacts poorly with old UNIX APIs...
      - jart 5y ago
chrsig 5y ago
I agree with you on all counts, but it's worth highlighting that the parent did not call the designers dumb, but rather the situation and implementation. I know it can seem like a distinction without a difference, but I think it's fair to critique work. I think they could have been more articulate and considerate towards the designers, and have some empathy that many people have worked very hard on it, and did their best in the problem/solution space they were working with. I just think it's important that people stay objective on if the quality of the people themselves is in question. It's not ok if it is, but I don't think it was here. now people that don't check how malloc() returns being blanketly labled as lazy is an example of it being about people. people don't ignore the possibility of a nullptr return from malloc because they're lazy. They ignore it because it's hard -- even if you do catch it, there's very little that you can actually do. You can't dynamically allocate...so I hope you've got enough stack space to do what you've gotta do. And even then, you have to be able to propagate that there's no memory all the way up the stack as it unwinds...and check every single allocation. The cpp world is mildly better in that it can throw std::bad_alloc...but if you have something that winds up doing an allocation in a destructor, I imagine that's not a fun time. Most of the time, there's not really a better thing to do than crash -- and there's not a lot of incentive to put any work into it. It's not something that should be happening on any sort of regular basis.
- andi999 5y ago
  But don't you need to check the return value of malloc to then reliable crash/terminate the program? It is like if your car runs out of gas you need to stop, but you don't abandon the steering wheel at high speed and hope for the best.
bitwize 5y ago
That's just the thing. fork(2), like much of POSIX, was a mistake. The proper semantics for starting a new process is something like posix_spawn or Win32 CreateProcess, i.e., you specify an executable image to start.
- jagrsw 5y ago
  > The proper semantics for starting a new process is.. :) Ah.. I guess when "I was younger" (TM) I was also so dogmatic on most things computer-related. Anyways, when we need to do something a bit more complex than equivalent of system(), then it quickly becomes evident, that in many cases we need to prepare ourselves for the future execve(). Here's the list of syscalls/c-funcs, which are called in two random projects I maintain, after fork() and before execve() (or execveat() or fexecve()). alarm(0); /* disable alarms */ setenv(); /* A couple of required envs, like MALLOC_PERTURB_ or MALLOC_PERTURB_ */ prctl(PR_SET_DUMPABLE, 1) /* regarding ptrace()-attach */ syscall(__NR_personality, ADDR_NO_RANDOMIZE); /* disable ASLR for debugging, if needed */ socketpair() /* reliable execve success detection, some form of witchcraft */ setpriority() prctl(PR_SET_PDEATHSIG, SIGKILL); /* die upon parents death */ setrlimit(); /* set of reset rlimits */ lseek(fd, 0, SEEK_SET); /* rewind input file for this specific subprocess */ /* prepare arguments (argv) for execve dynamically */ sysconf(_SC_NPROCESSORS_ONLN); pthread_setaffinity_np(); /* pin subprocess to a list of CPUs */ /* LOTS of functions here if we wanted to use net/process/mount namespacing e.g: assigning IP adddresses to interfaces creating custom views of the filesystem tree modifying capability sets */ open("/proc/self/oom_score_adj"), write(), close(); /* adjustment of oom score */ open("/proc/self/fd", O_DIRECTORY); getdents(); fcntl(F_GETFD); fcntl(F_SETFD, FD_CLOEXEC); close() /* closing fds upon exec */ setsid(); /* new session */ sigprocmask(empty_set); /* reset signal mask */ open("/dev/null"); dup2(null, 0..1); /* close fd 0,1,2 */ prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER); /* application of sandboxing */ and finally execv() or execveat() Granted, the projects are maintain are probably more on the heavy side of things, when it comes to process manipulation, before execve, but putting all of that in some control structure, would be down to impossible for me. Such structure would have to be so extensible, that it'd have to be some form of VM I guess effectively. So.. having ability to simply call a couple of syscalls from the context of a regular new process, and before execve() is quite good here. Sure.. maybe we should have some simple form of fork/execv, for those who want to call system() or popen() and not hit the memory overcommit related crashes. But not as a replacement, rather a new syscall. Even so, debugging while a process creation/execution failed would be madness, given that you simply would get EINVAL, and the failure could be related to any of dozen parameters in a process creation control structure.

giomasce 5y ago

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

gpderetta 5y ago
They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.
- OldHand2018 5y ago
  This paper [1] argues that fork needs to be deprecated in favor of posix_spawn() or other more modern solutions. It claims that, among many other reasons, that fork encourages overcommit and that programs such as Redis are extraordinarily constrained if overcommit is disabled - because of fork. [1] https://dl.acm.org/doi/10.1145/3317550.3321435 https://dl.acm.org/doi/10.1145/3317550.3321435
- scottlamb 5y ago
  > Vfork is usually a better solution. I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions." So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In practice, I believe it's fine, and on glibc posix_spawn is apparently written in terms of vfork, but libc is allowed to make assumptions that portable programs shouldn't. Story time: I once worked on a Unix-based system with no MMU. The fork implementation did something insane: it looked for things that were possibly pointers (four-byte aligned memory locations which can be interpreted as valid physical memory locations on this system allocated to that program) and adjusted them. It was kind of like a conservative GC in that it treated anything that looked like a pointer as if it were a pointer. But no reasonable person would call modifying memory that may or may not be pointers to be "conservative". Amazingly, it worked most of the time, but sometimes strings got corrupted, and as a workaround there were a bunch of places where string buffers were fully zeroed where just a NUL byte would otherwise do. This was early in my career, and I didn't design the system anyway. If I were working on it today I'd remove the fork implementation and make everything use vfork and/or posix_spawn instead. Apparently that's what posix_spawn was made for. Another weird problem with the lack of MMU: the compiler also didn't use register-based addressing (what's the term? like position-independent code but for the data segment?), so global variables were truly global, not just global to the process. A bunch of code needed to be "deglobalized" to deal with this. I wanted to improve the compiler, but long story short I got offered another job first.
  - pcwalton 5y ago
    No-MMU Linux usually just disables fork and uses vfork instead. The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI).
    - scottlamb 5y ago
      > The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI). Interesting, thanks. That didn't come up on our system—not only did we not use shared libraries but we also linked the whole system into one binary, kernel and all. Link times were atrocious in combination with identical code folding and big VLIW sentences, but it worked.
- jart 5y ago
  > They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. In other words, you write your C code inside a C string instead of in C.
joosters 5y ago
It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on… the flexibility of a separate fork() then exec() means you can set up the initial state of a new process exactly as you want, by doing whatever work is needed between the two calls. If you merge them into one, then you will never be able to encapsulate all of that.
- saagarjha 5y ago
  I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)
  - marcan_42 5y ago
    Are there any things that are illegal between fork and exec? It is perfectly legal to exec whenever you want, and it it perfectly legal to fork whenever you want. I'm not aware of any requirements around the fork/exec sequence.
    - tsimionescu 5y ago
      For a single-threaded process, sure. But in a multi threaded context, almost any operation done in the child after fork() can royally mess up the system. There are many versions of libcs where malloc() after fork() is likely to deadlock between the parent and child processes, as they share the internal malloc() locks.
      - marcan_42 5y ago
        Ah, yes, multithreading. That always interacts poorly with old UNIX APIs...
    - jart 5y ago

chrsig 5y ago

I agree with you on all counts, but it's worth highlighting that the parent did not call the designers dumb, but rather the situation and implementation. I know it can seem like a distinction without a difference, but I think it's fair to critique work. I think they could have been more articulate and considerate towards the designers, and have some empathy that many people have worked very hard on it, and did their best in the problem/solution space they were working with. I just think it's important that people stay objective on if the quality of the people themselves is in question. It's not ok if it is, but I don't think it was here. now people that don't check how malloc() returns being blanketly labled as lazy is an example of it being about people. people don't ignore the possibility of a nullptr return from malloc because they're lazy. They ignore it because it's hard -- even if you do catch it, there's very little that you can actually do. You can't dynamically allocate...so I hope you've got enough stack space to do what you've gotta do. And even then, you have to be able to propagate that there's no memory all the way up the stack as it unwinds...and check every single allocation. The cpp world is mildly better in that it can throw std::bad_alloc...but if you have something that winds up doing an allocation in a destructor, I imagine that's not a fun time. Most of the time, there's not really a better thing to do than crash -- and there's not a lot of incentive to put any work into it. It's not something that should be happening on any sort of regular basis.

andi999 5y ago
But don't you need to check the return value of malloc to then reliable crash/terminate the program? It is like if your car runs out of gas you need to stop, but you don't abandon the steering wheel at high speed and hope for the best.

bitwize 5y ago

That's just the thing. fork(2), like much of POSIX, was a mistake. The proper semantics for starting a new process is something like posix_spawn or Win32 CreateProcess, i.e., you specify an executable image to start.

jagrsw 5y ago
> The proper semantics for starting a new process is.. :) Ah.. I guess when "I was younger" (TM) I was also so dogmatic on most things computer-related. Anyways, when we need to do something a bit more complex than equivalent of system(), then it quickly becomes evident, that in many cases we need to prepare ourselves for the future execve(). Here's the list of syscalls/c-funcs, which are called in two random projects I maintain, after fork() and before execve() (or execveat() or fexecve()). alarm(0); /* disable alarms */ setenv(); /* A couple of required envs, like MALLOC_PERTURB_ or MALLOC_PERTURB_ */ prctl(PR_SET_DUMPABLE, 1) /* regarding ptrace()-attach */ syscall(__NR_personality, ADDR_NO_RANDOMIZE); /* disable ASLR for debugging, if needed */ socketpair() /* reliable execve success detection, some form of witchcraft */ setpriority() prctl(PR_SET_PDEATHSIG, SIGKILL); /* die upon parents death */ setrlimit(); /* set of reset rlimits */ lseek(fd, 0, SEEK_SET); /* rewind input file for this specific subprocess */ /* prepare arguments (argv) for execve dynamically */ sysconf(_SC_NPROCESSORS_ONLN); pthread_setaffinity_np(); /* pin subprocess to a list of CPUs */ /* LOTS of functions here if we wanted to use net/process/mount namespacing e.g: assigning IP adddresses to interfaces creating custom views of the filesystem tree modifying capability sets */ open("/proc/self/oom_score_adj"), write(), close(); /* adjustment of oom score */ open("/proc/self/fd", O_DIRECTORY); getdents(); fcntl(F_GETFD); fcntl(F_SETFD, FD_CLOEXEC); close() /* closing fds upon exec */ setsid(); /* new session */ sigprocmask(empty_set); /* reset signal mask */ open("/dev/null"); dup2(null, 0..1); /* close fd 0,1,2 */ prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER); /* application of sandboxing */ and finally execv() or execveat() Granted, the projects are maintain are probably more on the heavy side of things, when it comes to process manipulation, before execve, but putting all of that in some control structure, would be down to impossible for me. Such structure would have to be so extensible, that it'd have to be some form of VM I guess effectively. So.. having ability to simply call a couple of syscalls from the context of a regular new process, and before execve() is quite good here. Sure.. maybe we should have some simple form of fork/execv, for those who want to call system() or popen() and not hit the memory overcommit related crashes. But not as a replacement, rather a new syscall. Even so, debugging while a process creation/execution failed would be madness, given that you simply would get EINVAL, and the failure could be related to any of dozen parameters in a process creation control structure.