> If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction The report does look a lot like transparent hu

> If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction The report does look a lot like transparent hugepage defrag moving around memory randomly & system CPU spikes for no good reason. And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space. Literally the first thing I have to do to a Hadoop cluster is go around and turn off the defrag for the multi-threaded bits we have or system CPU goes over 20% as the system goes into workloads. The other way you end up with a thundering herd causing permanent issues is with NUMA balancing (+ false sharing). So if you scale up a system slowly, you end up allocating in the same NUMA zone for a whole section (like a 1Gb array), which makes traversing it faster, but if this happens at the same time as another thread touching the same data, it will get scheduled over in a different zone & the kernel can do a lot of busy work with zone rebalancing as you go above 50% memory usage. This used to be a big deal when the first ccNUMA Intel boxes were coming out & mysql at Yahoo used to have so much trouble with NUMA messing up memory access speed assumptions[1]. The 96 core + 235GiB of data suggests there is some NUMA messiness going on for sure, particularly because there's a pinned worker to a CPU in the design. The NUMA issues were front-and-center when the Power8 porting of Hadoop was going on , particularly because Java just gives you two things you can tweak directly in the GC (UseNUMA and UseTLAB). Because it was a lot of Cores on the bus (128 cores x 1TB - NUMA is like 1x-4x slower, which is hard to optimize for in a general sense). [1] - https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/ https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-...