10 ms·

Dropping cache didn’t drop cache

175 points by r4um 5y ago

flakiness 5y ago
Great read, although I don't think I have enough knowledge to fully appreciate the details of the article. I had similar problem before, and I didn't even notice that the cache was not cleared and had worked on pointless hypothesis until a coworker pointed out that there was a case where the kernel didn't evict the page cache. It's very hard to even detect that problem. Twitter's Engineering blog has several interesting posts recently btw. Kudos to them.
- bartvk 5y ago
  What I'm amazed, is that the patch is so small: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8199be001a470209f5c938570cc199abb012fe53 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - lilyball 5y ago
    It’s so small because it’s deleting a bad optimization.
    - vanderZwan 5y ago
      Ah yes, the old "Doctor, when I do [x] it hurts" "Then stop doing that" approach
- justaj 5y ago
  > Great read, although I don't think I have enough knowledge to fully appreciate the details of the article. Same here, but luckily I started reading the article expecting that. Primarily because: > The article requires some prerequisite knowledge about Linux kernel memory management subsystem. Oh how I wish all articles had this sort of disclaimer.
ketzo 5y ago
The saying I heard in my embedded programming class was: “There are two hard things in programming: naming things, cache invalidation, and off-by-one-errors.”
aeyes 5y ago
I am amazed that Twitter employs kernel developers.
- bartvk 5y ago
  The writer recently joined, if their Twitter bio is correct (August 2020). They have a handle with a bunch of numbers, and zero tweets: https://twitter.com/YangShi05755293 https://twitter.com/YangShi05755293 I couldn't really find there are more in the team.
  - saagarjha 5y ago
    A quick search shows that the team includes at least several more engineers, which is not very surprising.
- EnderWT 5y ago
  From this job posting[1]: "The Kernel & Operating System team at Twitter is responsible for shipping the Linux kernel and OS versions on which all of Twitter’s services run. Our job is to ensure the safe, reliable, fast, and efficient operation of the lowest levels of the Twitter software stack! We actively contribute many of our fixes upstream and are active members of the Linux OSS community." [1]https://careers.twitter.com/en/work-for-twitter/202101/005c4b36-7267-4cbd-9988-d4be72b0f10c/dbcd92fc-a9b4-4628-88a4-323152561545.html/engineering-manager-kernel-operating-systems.html https://careers.twitter.com/en/work-for-twitter/202101/005c4...
- saagarjha 5y ago
  Most large companies employ at least a handful of kernel developers to keep their servers running right. Some (Google, Facebook, etc.) have large teams that contribute significant amounts of code upstream, often based on what they're running internally.
- oblio 5y ago
  FAANG & co. discovered that not being afraid of software, but instead harnessing its power, lets them leave old school enterprises in the dust. And once you're no longer afraid of software, but instead an active creator of software, it's much cheaper and safer to instead employ some open source developers to take care of maintenance for your stuff. The old school enterprise way would have been to pay IBM, Red Hat, Oracle, whoever, millions and tens of millions of dollars in support for Cover-Your-A** type policies set by middle managers. FAANG & co. instead hire developers for this maintenance, and guess what, once you hire these people, they can do other things for you, because they're generally smart people (and also motivated, can work without supervision, etc.).
aetherspawn 5y ago
I know that Linus hates unit tests, but these kinds of scenarios are perfect for regression tests. The investigation of the issue took so long, that you'd really want to spend those extra few hours writing the test to save yourself investigation in the future. The patch doesn't include any comment about a race condition in the actual code, so let's assume that all the knowledge is more-or-less lost and forgotten in 12 months or so.
- enw 5y ago
  This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.
  - YZF 5y ago
    The answer to how is the kernel tested is (historically) mostly manually by a lot of people: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux... Basically what used to be known as alpha/beta testing. Seems like there's more automated testing these days. I work on a smaller piece of software with many automated tests that is significantly less reliable. I wonder if there's some lessons there. Another interesting tidbit is that our tests break a lot as well. In "The Olde Days" many a pieces of complex/reliable software was written without a single unit test or any automated test for that matter. Sure, some of it had bugs, maybe even nasty bugs, but so does most software today. Have we made progress? maybe.
    - josephg 5y ago
      Tests let you modify code that’s unfamiliar to you without fear you’ll accidentally break something. If you live in a codebase and work with it every day, automated testing is arguably less important because you build an intuition around what kind of changes might break things. I wouldn’t be surprised if the lack of testing has actively fostered the ownership model Linux has adopted. You would need something like that to keep the bugs at bay. But there’s lots of code out there which doesn’t (or can’t) have a dedicated owner in the same kind of way. I had an issue opened on one of my GitHub projects the other day. I’m the sole maintainer of this project, the code is fiendishly complex and it’s working without any changes for about 18 months. Somebody found a bug! But after 18 months I’ve forgotten the mental context I had while writing that code - so I was terrified I’d break something else while fixing that problem. But I had an extensive test suite to fall back on. So when I fixed the bug and the existing tests all passed, I had the confidence to publish a new version. (And of course, I added the bug’s repro to the test suite for next time). Lots of projects need guard rails like this. Maybe they’re solo projects. Or maybe there’s a lot of churn in the team. Sometimes there’s just more code than there are engineers to keep track of everything, so people are bouncing around a lot and don’t build much expertise in a single area. For the long tail of software, I’m pretty confident in saying unit tests have dramatically improved reliability beyond the standard we had a couple of decades ago. Have they made us lazier too? Maybe! But in my book they’re definitely a win on the whole.
bboreham 5y ago
I tried to debug a very similar-sounding issue a couple of years back. Many GB used in dentrys, not shrinking when asked, no obvious cause. Sadly I have no kernel hacking skills, don’t even know what a dentry is. Kudos to the author.
- a1369209993 5y ago
  FWIW, I'd assume that's "directory entry", the (file name,inode number) pair that's used to associate names of files in a directory with the data structures (inodes) describing those files. Not sure how you'd end with many gigabytes of them though, they're usually on the order of length-of-name + 8-24 bytes IIRC, so that would be around tens of millions of files?
  - bboreham 5y ago
    From TFA: “it seems that dentry caches consume around 2GB memory under one memory cgroup”. My issue happened after stress-testing Kubernetes by repeatedly starting and stopping many containers. So over time there could be tens of millions of files bind-mounted.
    - bboreham 5y ago
      Here is the output from 'slabtop' that I captured at the time, right after a commanded cache drop - looks like dentry wasn't the biggest component, but there were 32 million of them: OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 534740452 533096794 99% 0.03K 4312423 124 17249692K kmalloc-32 32734926 23946797 73% 0.19K 1558806 21 6235224K dentry 86560 86558 99% 64.00K 86560 1 5539840K kmalloc-65536 2124646 1994340 93% 2.00K 1062323 2 4249292K kmalloc-2048 1013664 1013606 99% 4.00K 1013664 1 4054656K kmalloc-4096 14592544 11935392 81% 0.12K 456017 32 1824068K kmalloc-128 1300528 1158942 89% 1.00K 325132 4 1300528K kmalloc-1024
say_it_as_it_is 5y ago
How does a Linux kernel team prioritize work when it's not investigating issues?
- reasonabl_human 5y ago
  Using a JIRA board, I suppose. Investigating issues would only be one facet of the job, other examples could be building optimizations for the low-level stack, evaluating upcoming releases, etc.

175 points by r4um 5y ago

Great read, although I don't think I have enough knowledge to fully appreciate the details of the article. I had similar problem before, and I didn't even notice that the cache was not cleared and had worked on pointless hypothesis until a coworker pointed out that there was a case where the kernel didn't evict the page cache. It's very hard to even detect that problem. Twitter's Engineering blog has several interesting posts recently btw. Kudos to them.

bartvk 5y ago
What I'm amazed, is that the patch is so small: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8199be001a470209f5c938570cc199abb012fe53 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- lilyball 5y ago
  It’s so small because it’s deleting a bad optimization.
  - vanderZwan 5y ago
    Ah yes, the old "Doctor, when I do [x] it hurts" "Then stop doing that" approach
justaj 5y ago
> Great read, although I don't think I have enough knowledge to fully appreciate the details of the article. Same here, but luckily I started reading the article expecting that. Primarily because: > The article requires some prerequisite knowledge about Linux kernel memory management subsystem. Oh how I wish all articles had this sort of disclaimer.

ketzo 5y ago

The saying I heard in my embedded programming class was: “There are two hard things in programming: naming things, cache invalidation, and off-by-one-errors.”

aeyes 5y ago

I am amazed that Twitter employs kernel developers.

bartvk 5y ago
The writer recently joined, if their Twitter bio is correct (August 2020). They have a handle with a bunch of numbers, and zero tweets: https://twitter.com/YangShi05755293 https://twitter.com/YangShi05755293 I couldn't really find there are more in the team.
- saagarjha 5y ago
  A quick search shows that the team includes at least several more engineers, which is not very surprising.
EnderWT 5y ago
From this job posting[1]: "The Kernel & Operating System team at Twitter is responsible for shipping the Linux kernel and OS versions on which all of Twitter’s services run. Our job is to ensure the safe, reliable, fast, and efficient operation of the lowest levels of the Twitter software stack! We actively contribute many of our fixes upstream and are active members of the Linux OSS community." [1]https://careers.twitter.com/en/work-for-twitter/202101/005c4b36-7267-4cbd-9988-d4be72b0f10c/dbcd92fc-a9b4-4628-88a4-323152561545.html/engineering-manager-kernel-operating-systems.html https://careers.twitter.com/en/work-for-twitter/202101/005c4...
saagarjha 5y ago
Most large companies employ at least a handful of kernel developers to keep their servers running right. Some (Google, Facebook, etc.) have large teams that contribute significant amounts of code upstream, often based on what they're running internally.
oblio 5y ago
FAANG & co. discovered that not being afraid of software, but instead harnessing its power, lets them leave old school enterprises in the dust. And once you're no longer afraid of software, but instead an active creator of software, it's much cheaper and safer to instead employ some open source developers to take care of maintenance for your stuff. The old school enterprise way would have been to pay IBM, Red Hat, Oracle, whoever, millions and tens of millions of dollars in support for Cover-Your-A** type policies set by middle managers. FAANG & co. instead hire developers for this maintenance, and guess what, once you hire these people, they can do other things for you, because they're generally smart people (and also motivated, can work without supervision, etc.).

aetherspawn 5y ago

I know that Linus hates unit tests, but these kinds of scenarios are perfect for regression tests. The investigation of the issue took so long, that you'd really want to spend those extra few hours writing the test to save yourself investigation in the future. The patch doesn't include any comment about a race condition in the actual code, so let's assume that all the knowledge is more-or-less lost and forgotten in 12 months or so.

enw 5y ago
This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.
- YZF 5y ago
  The answer to how is the kernel tested is (historically) mostly manually by a lot of people: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux... Basically what used to be known as alpha/beta testing. Seems like there's more automated testing these days. I work on a smaller piece of software with many automated tests that is significantly less reliable. I wonder if there's some lessons there. Another interesting tidbit is that our tests break a lot as well. In "The Olde Days" many a pieces of complex/reliable software was written without a single unit test or any automated test for that matter. Sure, some of it had bugs, maybe even nasty bugs, but so does most software today. Have we made progress? maybe.
  - josephg 5y ago
    Tests let you modify code that’s unfamiliar to you without fear you’ll accidentally break something. If you live in a codebase and work with it every day, automated testing is arguably less important because you build an intuition around what kind of changes might break things. I wouldn’t be surprised if the lack of testing has actively fostered the ownership model Linux has adopted. You would need something like that to keep the bugs at bay. But there’s lots of code out there which doesn’t (or can’t) have a dedicated owner in the same kind of way. I had an issue opened on one of my GitHub projects the other day. I’m the sole maintainer of this project, the code is fiendishly complex and it’s working without any changes for about 18 months. Somebody found a bug! But after 18 months I’ve forgotten the mental context I had while writing that code - so I was terrified I’d break something else while fixing that problem. But I had an extensive test suite to fall back on. So when I fixed the bug and the existing tests all passed, I had the confidence to publish a new version. (And of course, I added the bug’s repro to the test suite for next time). Lots of projects need guard rails like this. Maybe they’re solo projects. Or maybe there’s a lot of churn in the team. Sometimes there’s just more code than there are engineers to keep track of everything, so people are bouncing around a lot and don’t build much expertise in a single area. For the long tail of software, I’m pretty confident in saying unit tests have dramatically improved reliability beyond the standard we had a couple of decades ago. Have they made us lazier too? Maybe! But in my book they’re definitely a win on the whole.

bboreham 5y ago

I tried to debug a very similar-sounding issue a couple of years back. Many GB used in dentrys, not shrinking when asked, no obvious cause. Sadly I have no kernel hacking skills, don’t even know what a dentry is. Kudos to the author.

a1369209993 5y ago
FWIW, I'd assume that's "directory entry", the (file name,inode number) pair that's used to associate names of files in a directory with the data structures (inodes) describing those files. Not sure how you'd end with many gigabytes of them though, they're usually on the order of length-of-name + 8-24 bytes IIRC, so that would be around tens of millions of files?
- bboreham 5y ago
  From TFA: “it seems that dentry caches consume around 2GB memory under one memory cgroup”. My issue happened after stress-testing Kubernetes by repeatedly starting and stopping many containers. So over time there could be tens of millions of files bind-mounted.
  - bboreham 5y ago
    Here is the output from 'slabtop' that I captured at the time, right after a commanded cache drop - looks like dentry wasn't the biggest component, but there were 32 million of them: OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 534740452 533096794 99% 0.03K 4312423 124 17249692K kmalloc-32 32734926 23946797 73% 0.19K 1558806 21 6235224K dentry 86560 86558 99% 64.00K 86560 1 5539840K kmalloc-65536 2124646 1994340 93% 2.00K 1062323 2 4249292K kmalloc-2048 1013664 1013606 99% 4.00K 1013664 1 4054656K kmalloc-4096 14592544 11935392 81% 0.12K 456017 32 1824068K kmalloc-128 1300528 1158942 89% 1.00K 325132 4 1300528K kmalloc-1024

say_it_as_it_is 5y ago

How does a Linux kernel team prioritize work when it's not investigating issues?

reasonabl_human 5y ago
Using a JIRA board, I suppose. Investigating issues would only be one facet of the job, other examples could be building optimizations for the low-level stack, evaluating upcoming releases, etc.