3 ms·

Not just ubuntu. We tell our engineers to remove their swap partitions because sometimes the test suite will consume all remaining memory and push into swap. If

by tfha 9y ago

Not just ubuntu. We tell our engineers to remove their swap partitions because sometimes the test suite will consume all remaining memory and push into swap. If that happens, pretty much the only thing you can do is a hard poweroff. If you're fast, you can Ctrl+C and only lose a minute or two, but often you're just completely stuck. Much better to OOM and have the test suite killed early.

Matt3o12_ 9y ago
Wouldn’t it make much more sense to limit the maximum amount of memory available to something like (system memory-500MB) using CGroups? I know configuring CGroups can be difficult because there is no command line script but it might be worth it if it’s such a common problem with your test suits. An easy way to configure CGroups would be to use docker but that adds a lot of overhead in management (not in performance - but you need to have every dev install docker and test the suit using docker). I use swap quite a lot, though, I’m not sure what my system moves to swap though. A large portion seems to be inactive google chrome tabs (whenever I open an inactive tab, chrome freezes for a second and I suddenly use ~70MB less swap.
cat199 9y ago
> If that happens, pretty much the only thing you can do is a hard poweroff. Sounds like a non-swap system/kernel tuning issue - you should still be able to slowly switch to a virtual console or ssh in, etc, and kill the test suite.. Also, most linux distros run without any real usage of user/group resource limiting - setting up these appropriately is useful to limit the overall impact of runaway jobs
cosarara97 9y ago
Can ulimit help here?
- AstralStorm 9y ago
  Ulimit is per process, it won't do. vm.overcommit_memory=2 with some setting of vm.overcommit_ratio or bytes will help though. I'd say overcommit heuristics break applications and cause them to eat too much memory since they don't know when to stop. The only trouble is KVM which for some reason takes double of process space allocated to the VM for no good reason and perhaps memory intensive Java. You need to tune the latter anyway.

by tfha 9y ago

Matt3o12_ 9y ago

Wouldn’t it make much more sense to limit the maximum amount of memory available to something like (system memory-500MB) using CGroups? I know configuring CGroups can be difficult because there is no command line script but it might be worth it if it’s such a common problem with your test suits. An easy way to configure CGroups would be to use docker but that adds a lot of overhead in management (not in performance - but you need to have every dev install docker and test the suit using docker). I use swap quite a lot, though, I’m not sure what my system moves to swap though. A large portion seems to be inactive google chrome tabs (whenever I open an inactive tab, chrome freezes for a second and I suddenly use ~70MB less swap.

cat199 9y ago

> If that happens, pretty much the only thing you can do is a hard poweroff. Sounds like a non-swap system/kernel tuning issue - you should still be able to slowly switch to a virtual console or ssh in, etc, and kill the test suite.. Also, most linux distros run without any real usage of user/group resource limiting - setting up these appropriately is useful to limit the overall impact of runaway jobs

cosarara97 9y ago

Can ulimit help here?

AstralStorm 9y ago
Ulimit is per process, it won't do. vm.overcommit_memory=2 with some setting of vm.overcommit_ratio or bytes will help though. I'd say overcommit heuristics break applications and cause them to eat too much memory since they don't know when to stop. The only trouble is KVM which for some reason takes double of process space allocated to the VM for no good reason and perhaps memory intensive Java. You need to tune the latter anyway.