3 ms·

Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.

by Thaxll 8y ago

tayo42 8y ago
There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?
- devereaux 8y ago
  Better than turning it off, on Linux try to use vm.overcommit_memory It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra. Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs. If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance
- sitkack 8y ago
  > Also, to the writer, I think you want to turn off swap? Using kernel panics as an alerting mechanism is a brutal solution. Why not track page faults? Most orgs that I interact with track paging across the whole fleet.
- aybassiouny 8y ago
  Lucy, even in a healthy state, already have many page faults. They are just not on the hot path, e.g. logging. They don't affect latency much and nobody cares for them. Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.
ovi256 8y ago
Obviously there are different levels of monitoring maturity, just like with all aspects of engineering. The brave poster solved the issue, so he leveled up by his own.

Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.

by Thaxll 8y ago

Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.

tayo42 8y ago

There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?

devereaux 8y ago
Better than turning it off, on Linux try to use vm.overcommit_memory It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra. Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs. If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance
sitkack 8y ago
> Also, to the writer, I think you want to turn off swap? Using kernel panics as an alerting mechanism is a brutal solution. Why not track page faults? Most orgs that I interact with track paging across the whole fleet.
aybassiouny 8y ago
Lucy, even in a healthy state, already have many page faults. They are just not on the hot path, e.g. logging. They don't affect latency much and nobody cares for them. Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.

ovi256 8y ago

Obviously there are different levels of monitoring maturity, just like with all aspects of engineering. The brave poster solved the issue, so he leveled up by his own.