8 ms·

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable corre

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now.

pjc50 5y ago
But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)
- magicalhippo 5y ago
  > But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages. Of course, kinda hard to fix this now...
  - enedil 5y ago
    That also means that if you used seccomp to restrict mprotect syscall, the child process may never ever write again, which is an unfortunate design choice.
    - magicalhippo 5y ago
      Seccomp can check for the PROT_EXEC flag though, and let the others pass, no?
  - toast0 5y ago
    You'd presumably need the parent to also do a system call to mark pages read-write after forking, and that would need to include the stack pages, and that sounds like lots of fun; especially since sometimes library functions fork (although most of that is to fork/exec and more specific apis for that exist now). Not that it isn't loads of fun anyway, but woe betide thee who forks and threads and makes the other threads have a read-only stack. Probably you'd get people just marking as read-write and returning in sigsegv handlers, which I'm sure has great security properties... OTOH, at least there's an opportunity to deny the remap in the handler and get a decent crashdump from the program or a sliver of hope for managing the situation.
  - tsimionescu 5y ago
    > Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). This doesn't work because of the horrible fork/exec design. If I am a huge process and I want to run `ls`, I will first have to clone myself using fork(), which may trigger an OOM, even if the first action of my clone would have been exec(), ignoring all of that memory. I am always surprised that no one has added a sane 'spawn process' primitive to replace fork/exec. Especially since fork() without exec() only really works in single-threaded processes.
    - naniwaduni 5y ago
      We do have posix_spawn &c., which are expressly designed to be implementable even in situations where fork doesn't make sense.
      - tsimionescu 5y ago
        Interesting, I'd never learned about that. If I understand correctly, this actually used to be implemented as a fork()+exec() on many systems, though. It's actually vfork() that addressed the problem and allowed an actually useful implementation of posix_spawn in Linux at least.
  - pjc50 5y ago
    > Of course, kinda hard to fix this now.. Well, yes, if you can break backwards compatibility you can do anything. Except run all the existing software.
secondcoming 5y ago
Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.
- alexgartrell 5y ago
  Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing your way, but it'd be phenomenally more expensive. I know this because every thing we introduce to the hierarchy in practice makes computing phenomenally less expensive.
  - cturner 5y ago
    "We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.
    - pjc50 5y ago
      Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975 https://news.ycombinator.com/item?id=9653975 For ages cygwin had to emulate it manually by hand-copying the process.
    - throwawaylinux 5y ago
      Linux can prevent overcommit. And that's not the reason for much faster process creation than Windows as far as I know.
    - 5y ago
trhway 5y ago
>there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now. Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people not configuring virtual memory any more we get the situation you describe.
- angry_octet 5y ago
  Virtual memory is always on... If you actually hit swap the system effectively deadlocks anyway because the swap daemon is too dumb by being too fair. Persistent memory might change this but it needs so much work.
  - trhway 5y ago
    it was a throwback down the memory lane and the years before my first Linux in 1995 i had been dealing with Windows where configuration of swap was "configuring virtual memory", so i automatically used that term. > If you actually hit swap the system effectively deadlocks anyway that depends. In many cases in the past the options would be either with swap and thus slow or pretty much not at all. And again these days there is so much memory that there is always a way to avoid the swapping. Though i've met funny situations in recent years like when a several terabyte sized database process would get killed by the OOMKiller on a machine with overcommit left on and no swap configured.
    - angry_octet 5y ago
      OOM kill exists because swap is not being useful. A process failure is something you can deal with, livelock isn't. If you have an algorithm that works out of core then you'll never hit swap because you maintain the working set below memory limits.
pcwalton 5y ago
> wouldn't you just account the COW pages against the parent until they are copied? That's what Linux does. But how do you return ENOMEM when the copy does happen and now the system is out of memory? Memory writes don't return error codes. The best you could do is send a signal, which is exactly what the OOM killer does.
- deleted 5y ago
  [deleted]
- lokedhs 5y ago
  That's why you ensure you have enough swap to cover the worst case scenario. You as swap not because you need to actually use it, but on order to be able to guarantee there is enough memory available if the worst case scenario happens. In normal circumstances, swap should never really be utilised.
  - vbezhenar 5y ago
    > In normal circumstances, swap should never really be utilised. I don't agree. There're plenty of dormant virtual memory pages which will never be used. Keeping them in RAM is wasting precious resources.
  - Hendrikto 5y ago
    > You as swap not because you need to actually use it […] In normal circumstances, swap should never really be utilised. Swap is actually there so anonymous pages can be evicted, and is often used long before there is memory contention, during normal operation. Not having swap means that only file-backed pages can be evicted. During memory contention, this can cause thrashing. During "normal" operation, it degrades performance. Swap only being used when memory runs out, as kind of "emergency RAM", is a very widespread misunderstanding.

pjc50 5y ago

But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)

magicalhippo 5y ago
> But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages. Of course, kinda hard to fix this now...
- enedil 5y ago
  That also means that if you used seccomp to restrict mprotect syscall, the child process may never ever write again, which is an unfortunate design choice.
  - magicalhippo 5y ago
    Seccomp can check for the PROT_EXEC flag though, and let the others pass, no?
- toast0 5y ago
  You'd presumably need the parent to also do a system call to mark pages read-write after forking, and that would need to include the stack pages, and that sounds like lots of fun; especially since sometimes library functions fork (although most of that is to fork/exec and more specific apis for that exist now). Not that it isn't loads of fun anyway, but woe betide thee who forks and threads and makes the other threads have a read-only stack. Probably you'd get people just marking as read-write and returning in sigsegv handlers, which I'm sure has great security properties... OTOH, at least there's an opportunity to deny the remap in the handler and get a decent crashdump from the program or a sliver of hope for managing the situation.
- tsimionescu 5y ago
  > Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). This doesn't work because of the horrible fork/exec design. If I am a huge process and I want to run `ls`, I will first have to clone myself using fork(), which may trigger an OOM, even if the first action of my clone would have been exec(), ignoring all of that memory. I am always surprised that no one has added a sane 'spawn process' primitive to replace fork/exec. Especially since fork() without exec() only really works in single-threaded processes.
  - naniwaduni 5y ago
    We do have posix_spawn &c., which are expressly designed to be implementable even in situations where fork doesn't make sense.
    - tsimionescu 5y ago
      Interesting, I'd never learned about that. If I understand correctly, this actually used to be implemented as a fork()+exec() on many systems, though. It's actually vfork() that addressed the problem and allowed an actually useful implementation of posix_spawn in Linux at least.
- pjc50 5y ago
  > Of course, kinda hard to fix this now.. Well, yes, if you can break backwards compatibility you can do anything. Except run all the existing software.

secondcoming 5y ago

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.

alexgartrell 5y ago
Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing your way, but it'd be phenomenally more expensive. I know this because every thing we introduce to the hierarchy in practice makes computing phenomenally less expensive.
- cturner 5y ago
  "We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.
  - pjc50 5y ago
    Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975 https://news.ycombinator.com/item?id=9653975 For ages cygwin had to emulate it manually by hand-copying the process.
  - throwawaylinux 5y ago
    Linux can prevent overcommit. And that's not the reason for much faster process creation than Windows as far as I know.
  - 5y ago

trhway 5y ago

>there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now. Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people not configuring virtual memory any more we get the situation you describe.

angry_octet 5y ago
Virtual memory is always on... If you actually hit swap the system effectively deadlocks anyway because the swap daemon is too dumb by being too fair. Persistent memory might change this but it needs so much work.
- trhway 5y ago
  it was a throwback down the memory lane and the years before my first Linux in 1995 i had been dealing with Windows where configuration of swap was "configuring virtual memory", so i automatically used that term. > If you actually hit swap the system effectively deadlocks anyway that depends. In many cases in the past the options would be either with swap and thus slow or pretty much not at all. And again these days there is so much memory that there is always a way to avoid the swapping. Though i've met funny situations in recent years like when a several terabyte sized database process would get killed by the OOMKiller on a machine with overcommit left on and no swap configured.
  - angry_octet 5y ago
    OOM kill exists because swap is not being useful. A process failure is something you can deal with, livelock isn't. If you have an algorithm that works out of core then you'll never hit swap because you maintain the working set below memory limits.

pcwalton 5y ago

> wouldn't you just account the COW pages against the parent until they are copied? That's what Linux does. But how do you return ENOMEM when the copy does happen and now the system is out of memory? Memory writes don't return error codes. The best you could do is send a signal, which is exactly what the OOM killer does.

deleted 5y ago
[deleted]
lokedhs 5y ago
That's why you ensure you have enough swap to cover the worst case scenario. You as swap not because you need to actually use it, but on order to be able to guarantee there is enough memory available if the worst case scenario happens. In normal circumstances, swap should never really be utilised.
- vbezhenar 5y ago
  > In normal circumstances, swap should never really be utilised. I don't agree. There're plenty of dormant virtual memory pages which will never be used. Keeping them in RAM is wasting precious resources.
- Hendrikto 5y ago
  > You as swap not because you need to actually use it […] In normal circumstances, swap should never really be utilised. Swap is actually there so anonymous pages can be evicted, and is often used long before there is memory contention, during normal operation. Not having swap means that only file-backed pages can be evicted. During memory contention, this can cause thrashing. During "normal" operation, it degrades performance. Swap only being used when memory runs out, as kind of "emergency RAM", is a very widespread misunderstanding.