4 ms·

Maybe you could restart the app before peak of memory usage and OOM? If the app performance depends on OS cache hits then this e.g. process line will prevent it

Maybe you could restart the app before peak of memory usage and OOM? If the app performance depends on OS cache hits then this e.g. process line will prevent its long slow death: if total memory usage > 80% for 2 cycles then restart

yjftsjthsd-h 8y ago
I think that would only help when the problem is that your application is using too much memory in the first place, in which case the real solution is to fix your app so you're not leaking memory. But it looks like what's happening here is that something else on the system uses enough memory to trip the OOM killer, which case a preemptive restart won't really help.
- imhoguy 8y ago
  Absolutely. I think the app may have interesting challenges, like whatever some real-time stream processing could be introduced to avoid loading entire stuff into memory or even to skip disk IO. But I assume the OP did some effort/gain calculation which made him choose the existing path.
  - yawboakye 8y ago
    OP here. You're right. The processed outputs and the recorded tracks are expected to be available <= 2s after the stream ends. That forces our hand to not be able to use memory conservatively. For example, one way to avoid massive memory usage will be to record concurrently (this has small memory footprint since we flush to disk after the write buffer hits 512KB so for about 50 concurrent recordings we need ~25MB), but process and upload sequentially. The recorded audio tracks are big, and have to be loaded to memory since the other tools we use for processing can't work with file on disk yet. The <= 2s requirement is impossible to meet when things are done sequentially.
    - imhoguy 8y ago
      Check memory-mapped files, may possibly work for you https://godoc.org/golang.org/x/exp/mmap https://godoc.org/golang.org/x/exp/mmap
moosingin3space 8y ago
It seems like this would be easiest to do with a cgroup memory limit and supervisor process. In systemd or kubernetes, this should be pretty easy, IIRC.
yawboakye 8y ago
Good point. But that would be me taking over the work of the OOM killer. It's possible for the app to hit 80% memory usage without causing OOM, when there's no other expensive process starting or demanding more memory. That's why I defer to the OOM killer. Also if the app hits 80% memory usage it's probably at its peak activity. Restart and it will quickly climb to >= 80% again. It will be restart games at peak activity, which won't be good at all.

yjftsjthsd-h 8y ago

I think that would only help when the problem is that your application is using too much memory in the first place, in which case the real solution is to fix your app so you're not leaking memory. But it looks like what's happening here is that something else on the system uses enough memory to trip the OOM killer, which case a preemptive restart won't really help.

imhoguy 8y ago
Absolutely. I think the app may have interesting challenges, like whatever some real-time stream processing could be introduced to avoid loading entire stuff into memory or even to skip disk IO. But I assume the OP did some effort/gain calculation which made him choose the existing path.
- yawboakye 8y ago
  OP here. You're right. The processed outputs and the recorded tracks are expected to be available <= 2s after the stream ends. That forces our hand to not be able to use memory conservatively. For example, one way to avoid massive memory usage will be to record concurrently (this has small memory footprint since we flush to disk after the write buffer hits 512KB so for about 50 concurrent recordings we need ~25MB), but process and upload sequentially. The recorded audio tracks are big, and have to be loaded to memory since the other tools we use for processing can't work with file on disk yet. The <= 2s requirement is impossible to meet when things are done sequentially.
  - imhoguy 8y ago
    Check memory-mapped files, may possibly work for you https://godoc.org/golang.org/x/exp/mmap https://godoc.org/golang.org/x/exp/mmap

moosingin3space 8y ago

It seems like this would be easiest to do with a cgroup memory limit and supervisor process. In systemd or kubernetes, this should be pretty easy, IIRC.

yawboakye 8y ago

Good point. But that would be me taking over the work of the OOM killer. It's possible for the app to hit 80% memory usage without causing OOM, when there's no other expensive process starting or demanding more memory. That's why I defer to the OOM killer. Also if the app hits 80% memory usage it's probably at its peak activity. Restart and it will quickly climb to >= 80% again. It will be restart games at peak activity, which won't be good at all.