5 ms·

I agree, with the caveat in the case of THP to disable it by default. And then measure to prove it’s worth enabling. Or even better, set the setting to ‘madvise

by nvarsj 9y ago

I agree, with the caveat in the case of THP to disable it by default. And then measure to prove it’s worth enabling. Or even better, set the setting to ‘madvise’ and let applications decide whether they want huge pages or not. It baffles me that THP became enabled by default (is it? I think it’s only a default on RHEL distros?). It really screws up many expectations that applications might assume about memory behavior (like the page size). In the majority of cases, THP is a bad, bad idea and anyone with perf or devops experience will agree with this I think. Do you want to impose GC like pause characteristics to all processes on your box? And possibly double, triple, or 10x your memory usage? Enable THP then.

icebraining 9y ago
(is it? I think it’s only a default on RHEL distros?) I can confirm that neither my Ubuntu nor Debian servers have is in "always", they're either "madvise" or "never".
- aruggirello 9y ago
  On Ubuntu 14.04/amd64 I see: $ cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never I can also confirm that my Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-101-generic x86_64) droplet has the same setting.
  - icebraining 9y ago
    Quite curious, I'm running Ubuntu 16.04.3 LTS (4.11.0-14-generic x86_64) and have "madvise".
    - jlgaddis 9y ago
      The parent said "droplet" so I assume Digital Ocean. Unless you've installed the host yourself, from scratch, you can't be sure the option hasn't been changed. Makes me think that your setting is a default and his was set by Digital Ocean.
      - icebraining 9y ago
        Yes, I find it interesting that DO would change random kernel flags.
        aruggirello 9y ago
        I suspect settings actually depend on kernel version: 4.4 -> 'always', 4.11 -> 'madvise' Perhaps there are issues with 'madvise' in kernels prior to 4.11, so they chose 'always' rather than 'never'.
      - aruggirello 9y ago
        No, I have another Ubuntu 16.04 machine at home - same kernel version, same settings. He must have installed kernel 4.11 manually , because linux-image-generic currently pulls kernel 4.4.0-101-generic on 16.04; settings depend on kernel version.
  - tribaal 9y ago
    The author opened a bug in Ubuntu about this here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1703742 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1703742 It is in the process of being fixed (it should be set to madvise).
- glandium 9y ago
  FWIW, Debian 9 (stretch) has it set to "madvise", but my Debian unstable machine has it to "always". Looking further, I can see that /boot/config-4.12.0-1-amd64 has: # CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y and /boot/config-4.13.0-1-amd64 has: CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y # CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set So this is a recent change. Edit: The linux kernel source says the default is always (in mm/Kconfig), and that's been true since 2011. The debian package changelog says the change occurred in 4.13.4-1: * thp: Enable TRANSPARENT_HUGEPAGE_ALWAYS instead of TRANSPARENT_HUGEPAGE_MADVISE The reason is not given in the changelog itself, but it's given in the git log of the debian packaging: As advised by Andrea Arcangeli - since commit 444eb2a449ef "mm: thp: set THP defrag by default to madvise and add a stall-free defrag option" this will generally be best for performance. https://anonscm.debian.org/cgit/kernel/linux.git/commit/debian/changelog?id=611a8e67260e8b8190ab991206a3867681d6df91 https://anonscm.debian.org/cgit/kernel/linux.git/commit/debi... Edit 2: The mentioned commit (444eb2a449ef) dates back to 4.6, so presumably, at least some performance issues with transparent huge pages may be gone since that version of the kernel.
  - mrob 9y ago
    Interesting. I'm running Debian unstable, and recently my system would sometimes lock up under heavy memory pressure. I'm using VirtualBox, which has its own kernel module, so I can't be sure Linux itself is to blame, but the timing seems to coincide with when I switched to that kernel version. Maybe transparent hugepages uncovered a VirtualBox bug or even a kernel bug. And I care about worst case performance more than average performance, so I just now set it to "never".
disconnected 9y ago
> I think it’s only a default on RHEL distros? Since "enabled=always" is the kernel default value, anything that uses a stock kernel (example: Arch family) or has to build its own (Gentoo) will probably have it enabled by default. I just checked, and my Gentoo and Manjaro systems have it set to "enabled=always".
takeda 9y ago
It's enabled by default because it actually works fine in most cases, it has issues with certain workloads (databases, hadoop etc) where you'll do much better if you allocate Huge Pages (not to be confused with THP) region in advance. Anyway, recently they added new "defer" mode for defragmentation so THP doesn't try to defragment (the main cause of the slow down) upon allocation and instead it is triggering it in background (via kswapd and kcompactd). This is now set to be the default. I think it is available in RedHat/CentOS 7.3+
- dboreham 9y ago
  Depends what you mean by "works fine". imho any feature that can sent the kernel off into a tens of seconds dream state underneath my process is just unforgivable and totally broken. Definitely good to hear that this is being done out of band now.
  - takeda 9y ago
    I guess what I said is that most of the time latency is not as important as throughput. And in those scenarios it generally works fine. The best out of both worlds though (although then it requires more manual work) is pre-allocating HugaPages in advance and then let application use them (if the application supports it) or through libhugetlbfs (if it doesn't). Edit: changed hugetlbfs to libhugetlbfs, so it's easier to find how to do it with man libhugetlbfs

by nvarsj 9y ago

icebraining 9y ago

(is it? I think it’s only a default on RHEL distros?) I can confirm that neither my Ubuntu nor Debian servers have is in "always", they're either "madvise" or "never".

aruggirello 9y ago
On Ubuntu 14.04/amd64 I see: $ cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never I can also confirm that my Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-101-generic x86_64) droplet has the same setting.
- icebraining 9y ago
  Quite curious, I'm running Ubuntu 16.04.3 LTS (4.11.0-14-generic x86_64) and have "madvise".
  - jlgaddis 9y ago
    The parent said "droplet" so I assume Digital Ocean. Unless you've installed the host yourself, from scratch, you can't be sure the option hasn't been changed. Makes me think that your setting is a default and his was set by Digital Ocean.
    - icebraining 9y ago
      Yes, I find it interesting that DO would change random kernel flags.
      - aruggirello 9y ago
        I suspect settings actually depend on kernel version: 4.4 -> 'always', 4.11 -> 'madvise' Perhaps there are issues with 'madvise' in kernels prior to 4.11, so they chose 'always' rather than 'never'.
    - aruggirello 9y ago
      No, I have another Ubuntu 16.04 machine at home - same kernel version, same settings. He must have installed kernel 4.11 manually , because linux-image-generic currently pulls kernel 4.4.0-101-generic on 16.04; settings depend on kernel version.
- tribaal 9y ago
  The author opened a bug in Ubuntu about this here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1703742 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1703742 It is in the process of being fixed (it should be set to madvise).
glandium 9y ago
FWIW, Debian 9 (stretch) has it set to "madvise", but my Debian unstable machine has it to "always". Looking further, I can see that /boot/config-4.12.0-1-amd64 has: # CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y and /boot/config-4.13.0-1-amd64 has: CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y # CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set So this is a recent change. Edit: The linux kernel source says the default is always (in mm/Kconfig), and that's been true since 2011. The debian package changelog says the change occurred in 4.13.4-1: * thp: Enable TRANSPARENT_HUGEPAGE_ALWAYS instead of TRANSPARENT_HUGEPAGE_MADVISE The reason is not given in the changelog itself, but it's given in the git log of the debian packaging: As advised by Andrea Arcangeli - since commit 444eb2a449ef "mm: thp: set THP defrag by default to madvise and add a stall-free defrag option" this will generally be best for performance. https://anonscm.debian.org/cgit/kernel/linux.git/commit/debian/changelog?id=611a8e67260e8b8190ab991206a3867681d6df91 https://anonscm.debian.org/cgit/kernel/linux.git/commit/debi... Edit 2: The mentioned commit (444eb2a449ef) dates back to 4.6, so presumably, at least some performance issues with transparent huge pages may be gone since that version of the kernel.
- mrob 9y ago
  Interesting. I'm running Debian unstable, and recently my system would sometimes lock up under heavy memory pressure. I'm using VirtualBox, which has its own kernel module, so I can't be sure Linux itself is to blame, but the timing seems to coincide with when I switched to that kernel version. Maybe transparent hugepages uncovered a VirtualBox bug or even a kernel bug. And I care about worst case performance more than average performance, so I just now set it to "never".

disconnected 9y ago

> I think it’s only a default on RHEL distros? Since "enabled=always" is the kernel default value, anything that uses a stock kernel (example: Arch family) or has to build its own (Gentoo) will probably have it enabled by default. I just checked, and my Gentoo and Manjaro systems have it set to "enabled=always".

takeda 9y ago

It's enabled by default because it actually works fine in most cases, it has issues with certain workloads (databases, hadoop etc) where you'll do much better if you allocate Huge Pages (not to be confused with THP) region in advance. Anyway, recently they added new "defer" mode for defragmentation so THP doesn't try to defragment (the main cause of the slow down) upon allocation and instead it is triggering it in background (via kswapd and kcompactd). This is now set to be the default. I think it is available in RedHat/CentOS 7.3+

dboreham 9y ago
Depends what you mean by "works fine". imho any feature that can sent the kernel off into a tens of seconds dream state underneath my process is just unforgivable and totally broken. Definitely good to hear that this is being done out of band now.
- takeda 9y ago
  I guess what I said is that most of the time latency is not as important as throughput. And in those scenarios it generally works fine. The best out of both worlds though (although then it requires more manual work) is pre-allocating HugaPages in advance and then let application use them (if the application supports it) or through libhugetlbfs (if it doesn't). Edit: changed hugetlbfs to libhugetlbfs, so it's easier to find how to do it with man libhugetlbfs