Definitely there's a lack of shared wisdom about tuning parameters (along with other things in HPC, at least in the circles I see), and too much churn with Linu

Definitely there's a lack of shared wisdom about tuning parameters (along with other things in HPC, at least in the circles I see), and too much churn with Linux v. userland (e.g. cgroups). Actually, I was forgetting the stupidity of the memory cgroup invoking the OOMkiller, rather than giving ENOMEM, if you purely use the cgroup for memory accounting -- a sensible-looking change from OpenVZ was rejected. That means there's probably no indication about what's happened unless the job is correlated with the syslog message. As I don't get to do that stuff any more, I haven't looked for a way to hook it now. There's obviously a window in which it can fail, but the resource manager can track the job's PSS, as well as at least ulimiting data and stack. You certainly don't want the job to start paging, if you have swap -- that definitely wastes the resource. If you really don't want to limit jobs' memory use, the resource manager can at least adjust the oom_score.