4 ms·

Transparent hugepages seem mostly useless, but standard hugepages can be a lifesaver. I have a postgres server with 1TB RAM, and without using hugepages the mac

Transparent hugepages seem mostly useless, but standard hugepages can be a lifesaver. I have a postgres server with 1TB RAM, and without using hugepages the machine would regularly fill the page table causing an OOM. With a suitable number of 1GB hugepages assigned, it is rock solid.

pocketarc 4y ago
Interesting - would you be able to expand a bit on why hugepages makes a difference for Postgres? I have only the most basic knowledge of hugepages - picked up on it being a necessity for stable performance in VMs, but don't understand why it would matter for Postgres as well. And do you end up having to configure Postgres to use hugepages, or does it pick them up automatically?
- gladiatr72 4y ago
  It's the difference of configuring the kernel to pre-allocate the contiguous memory for huge page allocation vs. _transparent_ huge pages that requires the kernel compaction process. Postgresql needs to be reconfigured to use them properly. The shared_buffers setting needs to be adjusted to allocate in [huge page size] units.
  - pocketarc 4y ago
    I appreciate that, thanks for the explanation. I understand the difference between hugepages and transparent huge pages, I just don't understand how not using hugepages was impacting the parent's Postgres, or causing OOM errors. @menaerus actually has a great comment on this that digs deeper into it, looking forward to seeing what comes out of that thread as well.
    - gnfargbl 4y ago
      I gave a more detailed response on the post by 'menaerus, but to answer your question on config directly, you first need to enable the appropriate number of hugepages with a kernel boot parameter, e.g. hugepagesz=1G hugepages=940 and then add these config lines to postgresql.conf: huge_pages = on huge_page_size = 1GB
wongarsu 4y ago
A database server seems like the ideal use case for (real) huge pages: a single application with a low-ish number of threads on a machine with a huge amount of RAM. Lots of page table misses with normal sized pages, but very little memory wasted from huge pages and no memory contention from other software. Not coincidentally, Microsoft's primary use case for 1GB pages on Windows is MS SQL server.
- jiggawatts 4y ago
  > Microsoft's primary use case for 1GB pages on Windows is MS SQL server. It is automatically enabled, but only if you have the "Lock pages in memory" privilege assigned to the database engine service account... ... which the default SYSTEM account has enabled by default... ... but not if you set up a typical cluster with a domain user account. So in other words, the scenario where you would want this enabled the most -- large expensive enterprise clusters -- is where it is accidentally disabled with no warning. I use this as a free 10-20% performance boost. Add in a few other similar tuning settings and I can get a 50% pref improvement on just about any "enterprise" cluster without having to get clever. Turbo buttons are fun to press.
  - pocketarc 4y ago
    > Add in a few other similar tuning settings Have you ever written up some of the things you do to improve performance, or is that something you're unable to share?
    - eptcyka 4y ago
      And lose a source of income? This isn't how business is done in MS lands.
    - speed_spread 4y ago
      I'd assume the biggest turbo buttons are already on stackoverflow. Some may depend on specific database usage patterns.
    - Rimintil 4y ago
menaerus 4y ago
> machine would regularly fill the page table causing an OOM That's quite interesting. How big the page-table must be on your server to directly or indirectly cause an OOM? 4-level page-table data-structure can address 512^4 page-tables. A single page-table can contain 512 entries. And each page-table entry is 64 bytes large. Unless I missed something, this means that upper bound memory consumption of page-table data-structure is 512^4 * 64B and which is 4TB of RAM so I guess it is theoretically possible to OOM on a 1TB machine. What I don't understand is how huge-pages would help you mitigate this problem because 2MB huge-page entry will essentially be made up from 512 entries from a single table. Linux call them compound pages. And given that all those entries will be vacant, and in-use for that particular huge-page, upper bound memory consumption will remain the same as with 4K pages. It has been recognized that compound pages might be wasteful because all those 511 entries are basically pointing to a head entry [1], so unless you're running recentish version of kernel on your system, you wouldn't see this advantage. [1] https://lwn.net/Articles/839737/ https://lwn.net/Articles/839737/
- gnfargbl 4y ago
  This is a machine which is dedicated to postgres. For the particular workload (heavy update) and filesystem (ZFS), it appears from experimentation that optimal performance is achieved by dedicating most of the RAM to postgres' shared buffers, and not really relying on the filesystem cache at all. (As an aside, that might be unusual, the "standard" advice seems to be to set shared_buffers to a smaller value and to rely on the OS cache.) So, I have about 95% of the RAM assigned to postgres shared buffers. Without hugepages enabled, this was fine when the database started up, but as (presumably) the buffer cache became fragmented over time, the size of the page tables grew to a point where the machine ran out of RAM. I do have vm.overcommit_memory = 2 (never overcommit), but even so the OOM killer was still invoked because, well, the machine had no RAM left. I can't remember exactly how big the pagetables had gotten in this situation, but I think it was a few tens of GBs. I'm actually using 1GB hugepages, but in all honesty I'm not sure how Linux deals with them or how postgres claims them. From /proc/meminfo it looks like the page table is quite small, with the "Hugetlb" and "DirectMap1G" numbers both around 1TB.
  - 4y ago

pocketarc 4y ago

Interesting - would you be able to expand a bit on why hugepages makes a difference for Postgres? I have only the most basic knowledge of hugepages - picked up on it being a necessity for stable performance in VMs, but don't understand why it would matter for Postgres as well. And do you end up having to configure Postgres to use hugepages, or does it pick them up automatically?

gladiatr72 4y ago
It's the difference of configuring the kernel to pre-allocate the contiguous memory for huge page allocation vs. _transparent_ huge pages that requires the kernel compaction process. Postgresql needs to be reconfigured to use them properly. The shared_buffers setting needs to be adjusted to allocate in [huge page size] units.
- pocketarc 4y ago
  I appreciate that, thanks for the explanation. I understand the difference between hugepages and transparent huge pages, I just don't understand how not using hugepages was impacting the parent's Postgres, or causing OOM errors. @menaerus actually has a great comment on this that digs deeper into it, looking forward to seeing what comes out of that thread as well.
  - gnfargbl 4y ago
    I gave a more detailed response on the post by 'menaerus, but to answer your question on config directly, you first need to enable the appropriate number of hugepages with a kernel boot parameter, e.g. hugepagesz=1G hugepages=940 and then add these config lines to postgresql.conf: huge_pages = on huge_page_size = 1GB

wongarsu 4y ago

A database server seems like the ideal use case for (real) huge pages: a single application with a low-ish number of threads on a machine with a huge amount of RAM. Lots of page table misses with normal sized pages, but very little memory wasted from huge pages and no memory contention from other software. Not coincidentally, Microsoft's primary use case for 1GB pages on Windows is MS SQL server.

jiggawatts 4y ago
> Microsoft's primary use case for 1GB pages on Windows is MS SQL server. It is automatically enabled, but only if you have the "Lock pages in memory" privilege assigned to the database engine service account... ... which the default SYSTEM account has enabled by default... ... but not if you set up a typical cluster with a domain user account. So in other words, the scenario where you would want this enabled the most -- large expensive enterprise clusters -- is where it is accidentally disabled with no warning. I use this as a free 10-20% performance boost. Add in a few other similar tuning settings and I can get a 50% pref improvement on just about any "enterprise" cluster without having to get clever. Turbo buttons are fun to press.
- pocketarc 4y ago
  > Add in a few other similar tuning settings Have you ever written up some of the things you do to improve performance, or is that something you're unable to share?
  - eptcyka 4y ago
    And lose a source of income? This isn't how business is done in MS lands.
  - speed_spread 4y ago
    I'd assume the biggest turbo buttons are already on stackoverflow. Some may depend on specific database usage patterns.
  - Rimintil 4y ago

menaerus 4y ago

> machine would regularly fill the page table causing an OOM That's quite interesting. How big the page-table must be on your server to directly or indirectly cause an OOM? 4-level page-table data-structure can address 512^4 page-tables. A single page-table can contain 512 entries. And each page-table entry is 64 bytes large. Unless I missed something, this means that upper bound memory consumption of page-table data-structure is 512^4 * 64B and which is 4TB of RAM so I guess it is theoretically possible to OOM on a 1TB machine. What I don't understand is how huge-pages would help you mitigate this problem because 2MB huge-page entry will essentially be made up from 512 entries from a single table. Linux call them compound pages. And given that all those entries will be vacant, and in-use for that particular huge-page, upper bound memory consumption will remain the same as with 4K pages. It has been recognized that compound pages might be wasteful because all those 511 entries are basically pointing to a head entry [1], so unless you're running recentish version of kernel on your system, you wouldn't see this advantage. [1] https://lwn.net/Articles/839737/ https://lwn.net/Articles/839737/

gnfargbl 4y ago
This is a machine which is dedicated to postgres. For the particular workload (heavy update) and filesystem (ZFS), it appears from experimentation that optimal performance is achieved by dedicating most of the RAM to postgres' shared buffers, and not really relying on the filesystem cache at all. (As an aside, that might be unusual, the "standard" advice seems to be to set shared_buffers to a smaller value and to rely on the OS cache.) So, I have about 95% of the RAM assigned to postgres shared buffers. Without hugepages enabled, this was fine when the database started up, but as (presumably) the buffer cache became fragmented over time, the size of the page tables grew to a point where the machine ran out of RAM. I do have vm.overcommit_memory = 2 (never overcommit), but even so the OOM killer was still invoked because, well, the machine had no RAM left. I can't remember exactly how big the pagetables had gotten in this situation, but I think it was a few tens of GBs. I'm actually using 1GB hugepages, but in all honesty I'm not sure how Linux deals with them or how postgres claims them. From /proc/meminfo it looks like the page table is quite small, with the "Hugetlb" and "DirectMap1G" numbers both around 1TB.
- 4y ago