4 ms·
Nice problem solving. I'd classify the primary root cause as a kernel bug. It's good to make use of otherwise unused memory for caches, but not to the extent t
by jmull 9y ago
Nice problem solving.
I'd classify the primary root cause as a kernel bug. It's good to make use of otherwise unused memory for caches, but not to the extent that the caches grow so large they slow things down.
Secondarily, there's probably something wrong in a system where you have to constantly poll and attempt to access large numbers of files that don't exist. (But probably 100% of systems that do anything useful have at least some weird cruft like this somewhere in them at any given time, so I'm not judging.)
- tyingq 9y agoWeb servers do that, especially when spiders ask for urls that don't exist.
- jmull 9y agoThat's a good point, although I don't know why a spider that isn't specifically malicious (or I guess just badly malfunctioning) would request millions of different files that don't exist, which is what's needed to trigger this problem.
- tyingq 9y agoMillions would be unusual, but I could see it happening. An old domain that used to be an image host, for example.
- solatic 9y agoThe author notes that future kernels did decide to introduce limits that would've prevented the slowdowns from happening, but the customer was running an outdated kernel. That's what made the article disappointing for me. Do all this impressive in-kernel debugging just to find out that you should've upgraded your systems first. Sigh...
- jdmichal 9y agoBut the fix was to limit the cache based on process memory constraints. jmull's point is that if a cache is permitted to blow out so big that lookups are impacting performance, then that cache is not really serving its purpose in the first place.
- mcherm 9y agoOther fixes would have been possible. If the hash table could be resized when necessary there would never have been a performance problem.