3 ms·

Jemalloc can use separate arenas for different threads which I imagine mostly solves the futex congestion issue. Perhaps it introduces new ones?

by _flux 3y ago

gpderetta 3y ago
IIRC glibc default malloc doesn't use per-thread arenas as they would waste too much memory on programs spawning tens of thousands of threads, and glibc can't really make too many workload assumptions. Instead I think it uses a fixed pool of arenas and tries to minimize contention. These days on Linux, with restartable sequences, you can have true per-cpu arenas with zero contention. Not sure which allocator use them though.
- JonChesterfield 3y ago
  https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree/doc/man/rseq.2 https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree... Thank you, I didn't know about this one. An allocator that seems to use it at https://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html
  - jeffbee 3y ago
    Not only does tcmalloc use rseq, the feature was contributed to Linux by the tcmalloc authors, for this purpose, among other purposes.
- celrod 3y ago
  > As pressure from thread collisions increases, additional arenas are created via mmap to relieve the pressure. The number of arenas is capped at eight times the number of CPUs in the system (unless the user specifies otherwise, see mallopt), which means a heavily threaded application will still see some contention, but the trade-off is that there will be less fragmentation. https://sourceware.org/glibc/wiki/MallocInternals https://sourceware.org/glibc/wiki/MallocInternals So glibc's malloc will use up to 8x #CPUs arenas. If you have 10_000 threads, there is likely to be contention.
- yxhuvud 3y ago
  Glibc allocator is also notoriously prone to not giving back memory that could be given back to the OS when running several threads over a long time period. Reducing pool size helps a bit, but not sufficiently to make memory stable.

Jemalloc can use separate arenas for different threads which I imagine mostly solves the futex congestion issue. Perhaps it introduces new ones?

by _flux 3y ago

Jemalloc can use separate arenas for different threads which I imagine mostly solves the futex congestion issue. Perhaps it introduces new ones?

gpderetta 3y ago

IIRC glibc default malloc doesn't use per-thread arenas as they would waste too much memory on programs spawning tens of thousands of threads, and glibc can't really make too many workload assumptions. Instead I think it uses a fixed pool of arenas and tries to minimize contention. These days on Linux, with restartable sequences, you can have true per-cpu arenas with zero contention. Not sure which allocator use them though.

JonChesterfield 3y ago
https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree/doc/man/rseq.2 https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree... Thank you, I didn't know about this one. An allocator that seems to use it at https://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html
- jeffbee 3y ago
  Not only does tcmalloc use rseq, the feature was contributed to Linux by the tcmalloc authors, for this purpose, among other purposes.
celrod 3y ago
> As pressure from thread collisions increases, additional arenas are created via mmap to relieve the pressure. The number of arenas is capped at eight times the number of CPUs in the system (unless the user specifies otherwise, see mallopt), which means a heavily threaded application will still see some contention, but the trade-off is that there will be less fragmentation. https://sourceware.org/glibc/wiki/MallocInternals https://sourceware.org/glibc/wiki/MallocInternals So glibc's malloc will use up to 8x #CPUs arenas. If you have 10_000 threads, there is likely to be contention.
yxhuvud 3y ago
Glibc allocator is also notoriously prone to not giving back memory that could be given back to the OS when running several threads over a long time period. Reducing pool size helps a bit, but not sufficiently to make memory stable.