3 ms·
I've had a really bad run-in with transparent hugepage defragmentation. In a workload consisting of many small-ish reductions, my programme spent over 80% of it
by lorenzhs 9y ago
I've had a really bad run-in with transparent hugepage defragmentation. In a workload consisting of many small-ish reductions, my programme spent over 80% of its total running time in pageblock_pfn_to_page (this was on a 4.4 kernel, https://github.com/torvalds/linux/blob/v4.4/mm/compaction.c#L74-L115 https://github.com/torvalds/linux/blob/v4.4/mm/compaction.c#...) and a total of 97% of the total time in hugepage compaction kernel code. Disabling hugepage defrag with echo never > /sys/kernel/mm/transparent_hugepage/defrag lead to an instant 30x performance improvement.
There's been some work to improve performance (e.g. https://github.com/torvalds/linux/commit/7cf91a98e607c2f935dbcc177d70011e95b8faff https://github.com/torvalds/linux/commit/7cf91a98e607c2f935d... in 4.6) but I haven't tried if this fixes my workload.
- mlrtime 9y agoDid you try allocating hugepages statically at startup? This will also remove the fragmentation.
- lorenzhs 9y agoThe algorithm was implemented in a big data framework that handles the allocations, so I would have needed to significantly adapt its memory subsystem to change this. I've talked to the authors, though, and it's not easy to change. Easier to disable transparent hugepage defrag, especially when there's a paper deadline to meet :)