6 ms·
I've had bad luck with transparent hugepages on my Linux machines
- alexandrnikitin 4y agoI wrote a post on how to measure perf impact and pitfalls of THP some time ago. I hope it helps https://alexandrnikitin.github.io/blog/transparent-hugepages-measuring-the-performance-impact/ https://alexandrnikitin.github.io/blog/transparent-hugepages...
- geocar 4y agoHave you tried turning off THP and just using -XX:+UseLargePages? It always seems to do same-or-better for me and I don't have to worry about some other process accidentally stealing my (precious) huge pages.
- Witionsion 4y ago[flagged]
- vardump 4y ago4k pages are too small and 2 MB pages too large. I wish we x86 users had 64 kB pages. 4 kB pages reduce CPU performance due to frequent TLB misses and make virtual memory perform worse due to number of faults.
- mjevans 4y agoEven 1MB pages might be desirable for modern hardware. It would be really interesting if there were some charts ran against different types of workloads. E.G. games, databases, compiles, etc. More interesting still if TLP (Transparent Large Pages) could be set to a given size per... cgroup maybe?
- vardump 4y agoDifferent applications would certainly benefit from different page sizes. Use 1G pages, and boom, TLB misses and faults are just gone. But definitely not a good default choice! Any choice is a tradeoff. The larger the page size, the more physical memory is going to be wasted. On the other hand, I guess with the current NVMe hardware the effect on swap might not be too bad. 64 kB pages would mean the number of TLB misses and faults are just one sixteenth of the current status quo. Diminishing returns and all.
- Vogtinator 4y agoOn AArch64 you can have multiple combinations of mapping sizes: 4k/2MiB/1GiB 16KiB/32MiB 64KiB/512MiB/4TiB
- classichasclass 4y agoOn Power ISA the most common page sizes are 4K and 64K. My Raptor Talos II has 64K pages in Fedora. That said, enough useful "workstation" things assume 4K and break on 64K that I'm hopeful aarch64's use of 16K pages will get people to realize everything doesn't use the same page size. I see transparent hugepages as trying to have your cake and eat it too.
- gnufx 4y agoWhat things typically fail? (Nothing I've noticed on an HPC system running a few desktop-ish things under x2go.)
- classichasclass 4y agoFor awhile, Chromium was the biggest offender, and some places may still ship incompatible Electron binaries. I don't use Chrome but I'm told this is now fixed. Hangover also needs 4K, and some kernel modules assume it. Those are all off the top of my head but overall it's starting to get better.
- geocar 4y agoThere's a lot of benefits to going to 1g-pages, but treating them as 256 4k-pages seems almost always to be a mistake; All I ever hear is horror stories. Does anyone have a story of "good luck" with transparent hugepages?
- gnfargbl 4y agoTransparent hugepages seem mostly useless, but standard hugepages can be a lifesaver. I have a postgres server with 1TB RAM, and without using hugepages the machine would regularly fill the page table causing an OOM. With a suitable number of 1GB hugepages assigned, it is rock solid.
- pocketarc 4y agoInteresting - would you be able to expand a bit on why hugepages makes a difference for Postgres? I have only the most basic knowledge of hugepages - picked up on it being a necessity for stable performance in VMs, but don't understand why it would matter for Postgres as well. And do you end up having to configure Postgres to use hugepages, or does it pick them up automatically?
- gladiatr72 4y agoIt's the difference of configuring the kernel to pre-allocate the contiguous memory for huge page allocation vs. _transparent_ huge pages that requires the kernel compaction process. Postgresql needs to be reconfigured to use them properly. The shared_buffers setting needs to be adjusted to allocate in [huge page size] units.
- pocketarc 4y agoI appreciate that, thanks for the explanation. I understand the difference between hugepages and transparent huge pages, I just don't understand how not using hugepages was impacting the parent's Postgres, or causing OOM errors. @menaerus actually has a great comment on this that digs deeper into it, looking forward to seeing what comes out of that thread as well.
- gnfargbl 4y agoI gave a more detailed response on the post by 'menaerus, but to answer your question on config directly, you first need to enable the appropriate number of hugepages with a kernel boot parameter, e.g. hugepagesz=1G hugepages=940 and then add these config lines to postgresql.conf: huge_pages = on huge_page_size = 1GB
- praseodym 4y agoBesides ZFS, VMware Workstation on Linux also interacts badly with transparent huge pages: https://unix.stackexchange.com/questions/679706/kcompacd0-using-100-cpu-with-vmware-workstation-16 https://unix.stackexchange.com/questions/679706/kcompacd0-us...
- fps_doug 4y agoHad that problem too, and it's not the first time VMware was hit. My "tune for vmware" script has grown to three lines over the years: echo never > /sys/kernel/mm/transparent_hugepage/defrag echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag echo 1 > /proc/sys/vm/compaction_proactiveness # default 20
- hrgiger 4y agoI am recently studying hpc configs and the docs I follow most (amd,suse) recommends blocking thp by default which I was able to replicate performance impact myself as well, on the otherside standard hugepages brings lot of advantage, I do setup during kernel boot but didnt test properly, currently using 4GB. If you really wanna bypass page latency I recommend buying a cheap persistence memory, not all filesystems support yet , (ext4 and xfs ?) but with -o dax mount option you dont use pages, if data is temporary tmpfs with hugepages as far as I can see quite common usage but I also found kernel memblock argument by simulating non-existent persistent memory and mounting a filesystem with dax brings similar performance or even faster most of my cases yet still you wanna script each time boot mkfs/mount
- robaato 4y agoWith our application, THP enabled manifests as excessive system cpu. Have a nice graph of busy system where it dropped from 30% to 5% immediately.
- mnw21cam 4y ago> In the normal default kernel settings, this only happens for processes that use the madvise(2) system call to tell the kernel that a mmap()'d area of theirs is suitable for this. And everything makes sense now. I was having terrible terrible problems with Java and transparent huge pages. It only affected Java. I think this is because Java was the only thing we were running that would pre-allocate a large chunk of address space for the heap like this. I ended up disabling THP. However, on a more recent install, the problem magically went away, and I got the impression that the problem was fixed. The article is recently-written, but they're not using an old Linux install, are they?
- rwmj 4y agoConversely we found in a microbenchmark the other day that allowing THP more than doubled the speed. Note glibc lets you turn on and off THP per process which is pretty useful for benchmarking if it helps or hinders performance. $ hyperfine ' nbdkit -U - data "1 * 10737418240" --run exit ' Benchmark 1: nbdkit -U - data "1 * 10737418240" --run exit Time (mean ± σ): 3.658 s ± 0.049 s [User: 0.406 s, System: 3.242 s] Range (min … max): 3.576 s … 3.713 s 10 runs $ hyperfine ' GLIBC_TUNABLES=glibc.malloc.hugetlb=1 nbdkit -U - data "1 * 10 737418240" --run exit ' Benchmark 1: GLIBC_TUNABLES=glibc.malloc.hugetlb=1 nbdkit -U - data "1 * 10 737418240" --run exit Time (mean ± σ): 1.655 s ± 0.007 s [User: 0.299 s, System: 1.350 s] Range (min … max): 1.643 s … 1.666 s 10 runs
- zX41ZdbW 4y agoWe achieved more than 3 times speedup using "on-demand" transparent huge pages in ClickHouse[1] for a very narrow use-case: random access to a hash table that does not fit in the L3 cache but is not much larger. But there was a surprise... more than 10 times degradation of overall Linux server performance due to increased physical memory fragmentation after a few days in production: https://github.com/ClickHouse/ClickHouse/commit/60054d177c8bc190e8187c7c5f5e8495a1957a04 https://github.com/ClickHouse/ClickHouse/commit/60054d177c8b... It was seven years ago, and I hope that the Linux kernel has been improved. I will need to try "revert of revert" of this commit. These changes cannot be tested by microbenchmarks, and only production usage can show their actual impact. Also, we successfully use huge pages for text section of the executable, and it is beneficial for the stability of performance benchmarks due to lowering the number of iTLB misses. [1] ClickHouse - high-performance OLAP DBMS: https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/
- jeffbee 4y agoThis is why the main feature of a "hugepage aware allocator" is when and how to deallocate, to avoid fragmentation or exploding a collapsed page. https://google.github.io/tcmalloc/temeraire.html https://google.github.io/tcmalloc/temeraire.html
- 4y ago
- Artemiscardiac 4y agohey
- mickeyp 4y agoTHP are a quick win for people who have little control over how malloc behaves. Like Python and numpy, for instance. I did a lot of HPC modelling ranging from hundreds of GiB to TiB-sized RAM servers and THP was an instantaneous win over not using it. Later on I experimented with LD_PRELOAD_PATH and libhugetlbfs and while it did stabilise things even more and reduced time spent in the page table, it was not even several factors 'better' than THP. A large part of this really boils down to your performance needs. Deterministic modelling -> fairly stable and reproducible malloc patterns -> THP will probably work OK. If your memory usage spikes a lot then THP is probably more hassle than it's worth. The kernel will spend an eternity thrashing and kicking the page table and trying in vain to clean up after itself. I can see why people feel THP sucks under those conditions! If anything, the real crime here is how awful Linux is at HPC without a lot of tuning and careful tweaking. Throw NUMA into the mix, file caches and the OOMkiller and it feels like it never moved out of the 90s. Combine it with poorly-configured hypervisors and your performance will yo-yo and you'll spend an eternity trying to figure out why (ask me how I know...)
- basilgohar 4y agoWhen you say: the real crime here is how awful Linux is at HPC without a lot of tuning and careful tweaking To what are you making the comparison? Are *BSDs better? HP-Unix? AIX? Windows? What's the competition in this space that makes Linux look bad in HPC? Edit: Added "in HPC" at the end for clarification.
- jsjohnst 4y agoI’m curious if GP has any positive examples too, but did want to mention that being “the best of the awful” doesn’t make one not “awful”.
- pixl97 4y agoTypically being 'best of awful' means you have both a difficult problem and a very small dataset to work with where the solutions are not easily generalizable. You have to look at each and every HPC workload as a custom application where very small changes in your input and environment could have massive behavior changes in the application performance.
- sylware 4y agoYeah, it seems the "transparent" is not a very good idea in the end. I made a basic 2MiB page allocator recently, I know I will add explicit 1GiB/512GiB(4-levels page tables) somewhen in the future. But since it is rare, I may go "brutal" with 2MiB pages continuity scanning at allocation time, the alignment constraint lighten a lot that scanning.
- LinuxBender 4y agoTHP can boost performance on apps that actually allocate massive amounts of memory under specific conditions but it's also had some issues with memory leaks specific to THP that are hard to spot. Have these been fixed? I've not seen any mention of it. I have always disabled THP on everything, especially machines with high memory pressure. On systems that I entirely engineer and control I over-provision memory so it's less of an issue, but on systems that I have provisioned for others I could not prevent them from using all the memory and that is where THP would get people into trouble. I've tried to encourage people to use cgroups with memory limits to no avail. Our work-around was to set fairly high min_free based on a formula suggested by Redhat and set overcommit ratio to 0 to create a safety buffer zone. I should add this was during the 3.x and 4.x kernel versions. I have not done any performance testing with THP on 5.x and rather just disabled it out of habit.
- jeffbee 4y agoThey’re not leaks, it’s just that the slop at the end of a present page is potentially 512 times more than it would be with 4K pages.
- ckennelly 4y agoThe default setting of max_ptes_none is also problematic. On a stock kernel, it's 511. TCMalloc's docs recommend using max_ptes_none set to 0 for this reason: https://github.com/google/tcmalloc/blob/master/docs/tuning.md#system-level-optimizations https://github.com/google/tcmalloc/blob/master/docs/tuning.m... (Disclosure: I work on TCMalloc and authored the above doc.)
- dboreham 4y agoAlmost amusing to see this article in 2023. It must be > 15 years since I battled THP problems with large jvm heaps, after RH enabled the feature in RHEL. To be honest I thought that since everyone switched to Ubuntu and they had THP disabled by default, the whole thing had been lost to the mysts of time.
- gnufx 4y agoAs I commented on another huge pages item, you don't always want to allocate memory with malloc -- specifically putting your Fortran arrays on the stack. Note that in current Debian the THP default is "always". Also "PageTables" in /proc/meminfo hasn't been mentioned yet.
- jeffbee 4y agoSo the presence of the khugepaged task in the process list gives users something to blame, but it sounds like this person’s system was just out of memory and was trying to do direct reclaim in that task. The use of ZFS is sort of a red flag. That thing is full of leaks. In future cases it would be useful to look at the stats in /sys to see what it is doing. Under non-OOM conditions it’s not easy for it to use a lot of CPU because it defaults to scanning only small runs of pages and waiting ten seconds between scans.
- guardiangod 4y agoTransparent hugepage doesn't work well with VMWare Workstation. Everytime I install Workstation on a Linux machine, I would have to disable hugepage after khugepaged 100% of a CPU core and drag the VM into a molasses-like state. I believe the consensus is that Workstation makes lots of memory allocation and fragment the memory, causing the defragment-er to kick in. Vmware says won't fix lol.
- AtlasBarfed 4y agoI see there is some sort of "page merging" processing going on here, which screams "defragmentation / garbage collection" class of problems, which all seem to have bad edge cases that are not infrequently occurring. Is that the basic problem?
- withinboredom 4y agoI was really hoping someone would have made a smart joke about “well what did you expect to happen when you cover your computer in transparent sheets of paper.” But alas, there was no joke to be found here. But to get back on topic, I’ve also had some bad luck with these but that was mostly due to me making the mistake of using them for a database (I didn’t RTFM).