5 ms·

Curious Case of a 99.9% Latency Hike

65 points by myinnerbanjo 8y ago

loa-in-backup 8y ago
Nice, easy read. Light but to the point.
Thaxll 8y ago
Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.
- tayo42 8y ago
  There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?
  - devereaux 8y ago
    Better than turning it off, on Linux try to use vm.overcommit_memory It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra. Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs. If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance
  - sitkack 8y ago
    > Also, to the writer, I think you want to turn off swap? Using kernel panics as an alerting mechanism is a brutal solution. Why not track page faults? Most orgs that I interact with track paging across the whole fleet.
  - aybassiouny 8y ago
    Lucy, even in a healthy state, already have many page faults. They are just not on the hot path, e.g. logging. They don't affect latency much and nobody cares for them. Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.
- ovi256 8y ago
  Obviously there are different levels of monitoring maturity, just like with all aspects of engineering. The brave poster solved the issue, so he leveled up by his own.
conorh 8y ago
Nice insight into the tools, but wouldn't a simple memory monitor have shown the issue?
- aybassiouny 8y ago
  That is already tracked, but it is not very meaningful. We already have a lot of page faults (and paged memory), they are just not on the hot path.
tjungblut 8y ago
You could add an aliveness probe that makes sure the binary is paged in completely.
hinkley 8y ago
I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers. They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem". The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough space left for OS-level caches and especially for memory mapped files.
- aybassiouny 8y ago
  The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.
  - haneefmubarak 8y ago
    I think that this is something that I've noticed MacOS has generally handled well (at least since Lion, which is when I started noticing), among other memory subsystem things (being a little smarter about what to swap out, etc). However, the MacOS kernel seems to require a much higher amount of wired memory than the Linux kernel for similar workloads, so... I don't know if those are related, but it'd be interesting to figure out what MacOS does differently (their kernel is open sourced) and to see if that behavior could be adapted into the Linux kernel.
    - jsjohnst 8y ago
      > this is something that I've noticed MacOS has generally handled well Really? I’ve had the opposite experience. MacOS running out of memory and paging heavily almost always necessitates a reboot to get performance fully restored, even if you kill the memory hog (if you aren’t willing to deal with a sluggish machine for a while that is).
  - existencebox 8y ago
    Preemptive apologies for this glorified "it's nice to hear it's not just me" of a post. To let you skip the anecdote if you don't care, I'd be very curious if someone can answer _why_ this happens. Earlier this week I was running into Very Strange errors coming out of some deep libraries in one of my hobby services. Nothing was specifically failing but things were going very slow and vomiting a bit. I observed that I had over 25g of active Ram (Typically idle at ~8g) and some absurd amount of paged data too. Even after killing almost all non-system running processes I was sitting at ~20g total, including a 10g nonpaged pool and 6g process private. On reboot, and for the week since? 8g. (On Win10)

65 points by myinnerbanjo 8y ago

Nice, easy read. Light but to the point.

Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.

tayo42 8y ago
There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?
- devereaux 8y ago
  Better than turning it off, on Linux try to use vm.overcommit_memory It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra. Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs. If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance
- sitkack 8y ago
  > Also, to the writer, I think you want to turn off swap? Using kernel panics as an alerting mechanism is a brutal solution. Why not track page faults? Most orgs that I interact with track paging across the whole fleet.
- aybassiouny 8y ago
  Lucy, even in a healthy state, already have many page faults. They are just not on the hot path, e.g. logging. They don't affect latency much and nobody cares for them. Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.
ovi256 8y ago
Obviously there are different levels of monitoring maturity, just like with all aspects of engineering. The brave poster solved the issue, so he leveled up by his own.

conorh 8y ago

Nice insight into the tools, but wouldn't a simple memory monitor have shown the issue?

aybassiouny 8y ago
That is already tracked, but it is not very meaningful. We already have a lot of page faults (and paged memory), they are just not on the hot path.

tjungblut 8y ago

You could add an aliveness probe that makes sure the binary is paged in completely.

hinkley 8y ago

I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers. They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem". The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough space left for OS-level caches and especially for memory mapped files.

aybassiouny 8y ago
The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.
- haneefmubarak 8y ago
  I think that this is something that I've noticed MacOS has generally handled well (at least since Lion, which is when I started noticing), among other memory subsystem things (being a little smarter about what to swap out, etc). However, the MacOS kernel seems to require a much higher amount of wired memory than the Linux kernel for similar workloads, so... I don't know if those are related, but it'd be interesting to figure out what MacOS does differently (their kernel is open sourced) and to see if that behavior could be adapted into the Linux kernel.
  - jsjohnst 8y ago
    > this is something that I've noticed MacOS has generally handled well Really? I’ve had the opposite experience. MacOS running out of memory and paging heavily almost always necessitates a reboot to get performance fully restored, even if you kill the memory hog (if you aren’t willing to deal with a sluggish machine for a while that is).
- existencebox 8y ago
  Preemptive apologies for this glorified "it's nice to hear it's not just me" of a post. To let you skip the anecdote if you don't care, I'd be very curious if someone can answer _why_ this happens. Earlier this week I was running into Very Strange errors coming out of some deep libraries in one of my hobby services. Nothing was specifically failing but things were going very slow and vomiting a bit. I observed that I had over 25g of active Ram (Typically idle at ~8g) and some absurd amount of paged data too. Even after killing almost all non-system running processes I was sitting at ~20g total, including a 10g nonpaged pool and 6g process private. On reboot, and for the week since? 8g. (On Win10)