5 ms·

I just tried bsdiff, and can I complain a little? I tried to diff two 1 GB files, and within a few seconds bsdiff was using enough memory to lock up my Ubuntu d

I just tried bsdiff, and can I complain a little? I tried to diff two 1 GB files, and within a few seconds bsdiff was using enough memory to lock up my Ubuntu desktop. 10 minutes later I managed to get a text console and kill it. Maybe I should blame Ubuntu for letting a single process hose the system, but it was still annoying. Edit: bsdiff is documented to use memory 17x the file size.

mistermann 10y ago
Interesting, I had assumed this (runaway process and unable to do anything) only happens on Windows. (speaking in general)
- aviraldg 10y ago
  This happens to me quite often on Linux and I've always wondered if there's a way to fix it. Is there?
  - adrianN 10y ago
    man ulimit?
  - superflyguy 10y ago
    ssh to the box then kill the process.
    - derefr 10y ago
      I've often been unable to connect to SSH on a host for lack of available CPU/memory to spawn the SSH receiver process + the shell. It's especially a problem with AWS's tN.micro instances: when something thrashes them for long enough, they ratchet down to such a low CPU allocation that it will likely take hours before SSHD gets context-switched to enough times to start your process. By then, all sorts of timeouts will likely have killed your session.
      - i336_ 10y ago
        That's worth making persistent noise about, I think. Because you just described something that can be accurately described as a type of DoS. Considering the number of AWS credentials out there, I can see malicious users logging in, forkbombing the instance, getting it to throttle back, and then walking away whistling until the person with dashboard login wakes up and can reboot the machine.
  - prattmic 10y ago
    Usually this happens on Linux because an application is using too much memory and the kernel is spending tons of resources reclaiming memory to prevent a system OOM. The system can easily slow to a crawl as memory accesses and allocations are constantly triggering reclaim and causing the system to thrash. The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running. The best way to prevent this, if you know an application will use lots of memory, is to run it in a memory cgroup. With a memory cgroup you can limit the applications total allowed memory, reserving some for the rest of the system. Unfortunately, I don't know of a super nice CLI, but here's an example using cgcreate/cgexec (https://packages.debian.org/stretch/cgroup-tools https://packages.debian.org/stretch/cgroup-tools): $ # Create a memory cgroup named 'mygroup', allowing $USER to change the parameters and run tasks in the cgroup. $ sudo cgcreate -a $USER -t $USER -g memory:mygroup $ # Set the memory limit to 10MB $ echo 10000000 > /sys/fs/cgroup/memory/mygroup/memory.limit_in_bytes $ # Run a program in the cgroup $ cgexec -g memory:mygroup ls
    - euyyn 10y ago
      > The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running. This is one of the problems multitask OS's were invented to solve. The answer is always let other tasks run too.
derefr 10y ago
> Maybe I should blame Ubuntu for letting a single process hose the system Speaking of: is there a good way to ask an OS to "reserve memory and CPU-time" for emergency interactive maintenance (e.g. spawning a console on a thrashing box), in a similar way to how ext2/3/4 "reserves space" for the root user for emergency maintenance? The only real way I can think of to do this (that doesn't involve already knowing what processes could thrash and cgroup'ing them) is to run a hypervisor and treat the dom0 as the emergency console. That still doesn't let you get inside the thrashing instance to fix it, though.
- dugmartin 10y ago
  It won't help with memory usage but you can nice a process to reduce its share of time in the process scheduler.
- chousuke 10y ago
  If you use systemd you can assign services to slices which can be assigned cgroup resource reservations / limits. I don't think any distro does too much with this out of the box yet (IIRC by default all services run under a single "system" slice, and user sessions under a user slice), but it should be possible to run an "emergency environment" slice in which the processes are always guaranteed some small amount of resources necessary to recover.
  - JdeBP 10y ago
    The LinkedIn people report that it's not that simple. * https://engineering.linkedin.com/blog/2016/08/don_t-let-linux-control-groups-uncontrolled https://engineering.linkedin.com/blog/2016/08/don_t-let-linu...
- y4mi 10y ago
  isn't "Ctrl + Alt + F1" to F7 essentially that? (on linux)
brink 10y ago
> I just tried bsdiff, and can I complain a little? It's HN. Do you even have to ask?

mistermann 10y ago

Interesting, I had assumed this (runaway process and unable to do anything) only happens on Windows. (speaking in general)

aviraldg 10y ago
This happens to me quite often on Linux and I've always wondered if there's a way to fix it. Is there?
- adrianN 10y ago
  man ulimit?
- superflyguy 10y ago
  ssh to the box then kill the process.
  - derefr 10y ago
    I've often been unable to connect to SSH on a host for lack of available CPU/memory to spawn the SSH receiver process + the shell. It's especially a problem with AWS's tN.micro instances: when something thrashes them for long enough, they ratchet down to such a low CPU allocation that it will likely take hours before SSHD gets context-switched to enough times to start your process. By then, all sorts of timeouts will likely have killed your session.
    - i336_ 10y ago
      That's worth making persistent noise about, I think. Because you just described something that can be accurately described as a type of DoS. Considering the number of AWS credentials out there, I can see malicious users logging in, forkbombing the instance, getting it to throttle back, and then walking away whistling until the person with dashboard login wakes up and can reboot the machine.
- prattmic 10y ago
  Usually this happens on Linux because an application is using too much memory and the kernel is spending tons of resources reclaiming memory to prevent a system OOM. The system can easily slow to a crawl as memory accesses and allocations are constantly triggering reclaim and causing the system to thrash. The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running. The best way to prevent this, if you know an application will use lots of memory, is to run it in a memory cgroup. With a memory cgroup you can limit the applications total allowed memory, reserving some for the rest of the system. Unfortunately, I don't know of a super nice CLI, but here's an example using cgcreate/cgexec (https://packages.debian.org/stretch/cgroup-tools https://packages.debian.org/stretch/cgroup-tools): $ # Create a memory cgroup named 'mygroup', allowing $USER to change the parameters and run tasks in the cgroup. $ sudo cgcreate -a $USER -t $USER -g memory:mygroup $ # Set the memory limit to 10MB $ echo 10000000 > /sys/fs/cgroup/memory/mygroup/memory.limit_in_bytes $ # Run a program in the cgroup $ cgexec -g memory:mygroup ls
  - euyyn 10y ago
    > The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running. This is one of the problems multitask OS's were invented to solve. The answer is always let other tasks run too.

derefr 10y ago

> Maybe I should blame Ubuntu for letting a single process hose the system Speaking of: is there a good way to ask an OS to "reserve memory and CPU-time" for emergency interactive maintenance (e.g. spawning a console on a thrashing box), in a similar way to how ext2/3/4 "reserves space" for the root user for emergency maintenance? The only real way I can think of to do this (that doesn't involve already knowing what processes could thrash and cgroup'ing them) is to run a hypervisor and treat the dom0 as the emergency console. That still doesn't let you get inside the thrashing instance to fix it, though.

dugmartin 10y ago
It won't help with memory usage but you can nice a process to reduce its share of time in the process scheduler.
chousuke 10y ago
If you use systemd you can assign services to slices which can be assigned cgroup resource reservations / limits. I don't think any distro does too much with this out of the box yet (IIRC by default all services run under a single "system" slice, and user sessions under a user slice), but it should be possible to run an "emergency environment" slice in which the processes are always guaranteed some small amount of resources necessary to recover.
- JdeBP 10y ago
  The LinkedIn people report that it's not that simple. * https://engineering.linkedin.com/blog/2016/08/don_t-let-linux-control-groups-uncontrolled https://engineering.linkedin.com/blog/2016/08/don_t-let-linu...
y4mi 10y ago
isn't "Ctrl + Alt + F1" to F7 essentially that? (on linux)

brink 10y ago

> I just tried bsdiff, and can I complain a little? It's HN. Do you even have to ask?