3 ms·

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ? I think you're getting warm. It's hard to know anything from this confu

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ? I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent huge pages in the background (and struggling to find/make unfragmented physical address space). The behavior change was due to the mmap size change they observed: before with the extra bytes the kernel didn't initially use huge pages. After, it did, so no background operation was necessary. (Or maybe MAP_SHARED helped because they had them backed by a non tmpfs filesystem, where huge issues aren't supported. Or maybe some behavior changes were just because they tested multiple times without rebooting and the physical page space got fragmented. Again, hard to know much from this article.) If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction. Then I'd switch to using dedicated huge pages, because huge pages really do speed things up by like 15% for a memory heavy workload; they're worth it if you can avoid THP's sometimes pathological behavior. Edit to add: Here is a much clearer article explaining the same problem. https://pingcap.com/blog/why-we-disable-linux-thp-feature-for-databases https://pingcap.com/blog/why-we-disable-linux-thp-feature-fo... I don't agree with the pingcap folks that you should permanently disable THP altogether but they did good debugging.

gopalv 5y ago
> If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction The report does look a lot like transparent hugepage defrag moving around memory randomly & system CPU spikes for no good reason. And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space. Literally the first thing I have to do to a Hadoop cluster is go around and turn off the defrag for the multi-threaded bits we have or system CPU goes over 20% as the system goes into workloads. The other way you end up with a thundering herd causing permanent issues is with NUMA balancing (+ false sharing). So if you scale up a system slowly, you end up allocating in the same NUMA zone for a whole section (like a 1Gb array), which makes traversing it faster, but if this happens at the same time as another thread touching the same data, it will get scheduled over in a different zone & the kernel can do a lot of busy work with zone rebalancing as you go above 50% memory usage. This used to be a big deal when the first ccNUMA Intel boxes were coming out & mysql at Yahoo used to have so much trouble with NUMA messing up memory access speed assumptions[1]. The 96 core + 235GiB of data suggests there is some NUMA messiness going on for sure, particularly because there's a pinned worker to a CPU in the design. The NUMA issues were front-and-center when the Power8 porting of Hadoop was going on , particularly because Java just gives you two things you can tweak directly in the GC (UseNUMA and UseTLAB). Because it was a lot of Cores on the bus (128 cores x 1TB - NUMA is like 1x-4x slower, which is hard to optimize for in a general sense). [1] - https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/ https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-...
- scottlamb 5y ago
  Great point about NUMA. It sounds like one request is basically "scan the entire gallery for matches to this image". This is similar to websearch. The ideal thing would be for each of the pinned, per-core threads to allocate and initialize its own memory region and be responsible for scanning just that memory region, so the 96 threads all work on the same request then move on to the next. This would be more efficient as well as have lower latency at <100% utilization. I don't think they're doing that or they wouldn't have talked about increasing concurrency "from 10, 20, 40, 60, 80 and 96". > And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space. Also tmpfs, fwiw. https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... says "Currently THP only works for anonymous memory mappings and tmpfs/shmem. But in the future it can expand to other filesystems." I'm eagerly waiting for that future... I don't think they're using tmpfs because they said "The default tmpfs on the host was untouched and was left at 50% (128 GiB)", and 235 GiB doesn't fit in 128 GiB.

gopalv 5y ago

> If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction The report does look a lot like transparent hugepage defrag moving around memory randomly & system CPU spikes for no good reason. And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space. Literally the first thing I have to do to a Hadoop cluster is go around and turn off the defrag for the multi-threaded bits we have or system CPU goes over 20% as the system goes into workloads. The other way you end up with a thundering herd causing permanent issues is with NUMA balancing (+ false sharing). So if you scale up a system slowly, you end up allocating in the same NUMA zone for a whole section (like a 1Gb array), which makes traversing it faster, but if this happens at the same time as another thread touching the same data, it will get scheduled over in a different zone & the kernel can do a lot of busy work with zone rebalancing as you go above 50% memory usage. This used to be a big deal when the first ccNUMA Intel boxes were coming out & mysql at Yahoo used to have so much trouble with NUMA messing up memory access speed assumptions[1]. The 96 core + 235GiB of data suggests there is some NUMA messiness going on for sure, particularly because there's a pinned worker to a CPU in the design. The NUMA issues were front-and-center when the Power8 porting of Hadoop was going on , particularly because Java just gives you two things you can tweak directly in the GC (UseNUMA and UseTLAB). Because it was a lot of Cores on the bus (128 cores x 1TB - NUMA is like 1x-4x slower, which is hard to optimize for in a general sense). [1] - https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/ https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-...

scottlamb 5y ago
Great point about NUMA. It sounds like one request is basically "scan the entire gallery for matches to this image". This is similar to websearch. The ideal thing would be for each of the pinned, per-core threads to allocate and initialize its own memory region and be responsible for scanning just that memory region, so the 96 threads all work on the same request then move on to the next. This would be more efficient as well as have lower latency at <100% utilization. I don't think they're doing that or they wouldn't have talked about increasing concurrency "from 10, 20, 40, 60, 80 and 96". > And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space. Also tmpfs, fwiw. https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... says "Currently THP only works for anonymous memory mappings and tmpfs/shmem. But in the future it can expand to other filesystems." I'm eagerly waiting for that future... I don't think they're using tmpfs because they said "The default tmpfs on the host was untouched and was left at 50% (128 GiB)", and 235 GiB doesn't fit in 128 GiB.