6 ms·
Default musl allocator considered harmful to performance
- burntcaramel 1y agoInstead of “harmful to performance”, why can’t we say “slow”? Harmful should be reserved for things that affect security or privacy e.g. accidentally encourage bugs like goto does.
- dwattttt 1y agoOnly once we convince C developers that a lack of performance isn't inherently harmful.
- DaSHacka 1y agoMore like Python / JS devs C devs are the few I've met that seem to actually care.
- amonith 1y agoI think you misunderstood. That's exactly the problem. C developers consider slow performance harmful, which is often dumb.
- 1718627440 1y agoExcept that's why I use hundreds of C programs every day, but complain about the few python programs and all the sloppy websites.
- amonith 1y agoYou do you. Most people don't care about software that much in general. The most important thing is that it does the job and it does it securely. C won't help you with bugs in any shape or form (in fact it's famously bug-friendly), so it often makes more sense to use a tech stack that either helps with those or lowers the cost on the developer side.
- dijit 1y agoexternalising developer cost onto runtime performance only makes sense if humans will spend more time writing than running (in aggregate). Essentially you’re telling me that the software being made is not useful to many people; because the cost of writing the software (a handful of developers) will spend more time writing the software than their userbase will in executing their software. Otherwise you’re inflicting something on humanity. Dumping toxic waste in a river is much cheaper than properly disposing of it too; yet we understand that we are causing harm to the environment and litigate people who do that. Slow software is fine in low volumes (think: shitting in the woods) but dumping it on huge numbers of users by default is honestly ridiculous (Teams, I’m looking at you: with your expectation to run always and on everyones machine!)
- 1718627440 1y ago> Most people don't care about software that much in general. This is an example of not caring about the software per se, but only about the outcome. > [C is] in fact it's famously bug-friendly Yes, but as a user I like that. I have a game that from the user-experience seams to have tons of use-after-free bugs. You see that as a user, as strings shown in the UI suddenly turn to garbage and then change very fast. Even with such fatal bugs, the program continues to work, which I like as a user, since I just want to play the game, I don't care if the program is correct. When I want to get rid of these garbage text, I simply close the in-game window and reopen it and everything is fine. On the other side there are games written in Pascal or Java, which might not have that much bugs, but every single null pointer exception is fatal. This led to me not playing the games anymore, because being good and then having the program crash is so frustrating. I rather have it running a bit longer with silent corruption.
- denotational 1y agoA null-pointer dereference in C will be just as fatal (modulo optimizations).
- kelnos 1y agoI think people also care that software runs reasonably quickly. Among non-technical people, "my Windows is slow" seems to be a common complaint.
- tialaramex 1y agoA language community which so prizes the linked list is in no position to go throwing such stones. Linux lucked out, when you're doing tricky wait free concurrent algorithms that intrusive linked list you hand designed was a good choice. But over in userland you'll find another hand rolled list in somebody's single threaded file parser and oh, the growable array would be fifty times faster, shame the C programmer doesn't have one in their toolbox.
- jayd16 1y ago"Considered harmful" is a meme they're referencing but yeah...its pretty stale at this point.
- Agingcoder 1y agoTo me it’s not a meme, it’s a reference to a very famous letter by dijkstra regarding goto statements. https://en.wikipedia.org/wiki/Considered_harmful https://en.wikipedia.org/wiki/Considered_harmful
- tialaramex 1y agoNot really "goto statements" so much as the go-to arbitrary control flow semantic aka jump. C's goto is a housecat to the full blown jump's tiger. No doubt an angry housecat is a nuisance but the tiger is much more dangerous. C goto won't let you jump straight into the middle of unrelated code, for example, but the jump instruction has no such limit and neither did the feature Dijkstra was discussing.
- jayd16 1y agoThat is the meme.
- bmn__ 1y agoYou want to object because of a misunderstanding. The usage of the word meme here is correct in its original sense. The word cliche would also work.
- sim7c00 1y agomaybe its not 'slow' but more 'generalized for a wide range of use-cases'? - because is it really slow for what it does, or simply slower compared to a specialized implementation? (this is calling a regular person car slow compared to an F1 car... sure the thing is fast but good luck takin ur kids on holiday or doing weekly shopping runs?)
- masklinn 1y ago“Generalised to a wide range of use cases” is a really strange way to say “unsuitable to most multi-threaded programs”. In 2025 an allocator not cratering multi-threaded programs is the opposite of specialisation.
- flohofwoe 1y agoIt only matters when your threads allocate with such a high frequency that they run into contention. A too high access frequency to a shared resource is not a "general case", but simply poorly designed multithreaded code (but besides, a high allocation frequency through the system allocator is also poor design for any single-threaded code, application code simply should not assume any specific performance behaviour from the system allocator).
- Joker_vD 1y agoWell, what is "such a high frequency"? Different allocators have different breaking points, and the musl's one is apparently very low. > application code simply should not assume any specific performance behaviour from the system allocator Technically, yes. Practically, no; that's why e.g. C++ standard mandates time complexity of its containers. If you can't assume any specific performance from your system, that means you have to prepare for every system-provided functionality to be exponentially slow and obviously you can't do that. Take, for instance, the JSON parser in GTA V [0]: apparently, sscanf(buffer, "%d", &n) calls strlen(buffer) internally, so using it to parse numbers in a hot loop on 2 MiB-long JSON craters your performance. On one hand, sure, one can argue that glibc/musl developers are within their right to implement sscanf however inefficiently they want, and the application developers should not expect any performance targets from it, and therefore, probably should not use it. On the other hand, what is even the point of the standard library if you're not supposed to use it for anything practical? Or, for that matter, why waste your time writing an implementation that no-one should use for anything practical anyhow, due to its abysmal performance? [0] https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
- userbinator 1y agoI believe musl is supposed to be optimised heavily for size, not speed.
- stonogo 1y agoSpecifically its goals are low memory overhead and hardening. Safe defaults, and easy to swap to a performance-oriented malloc for those apps that want it. My question is: why is Rust performance contingent on a C malloc?
- masklinn 1y ago> why is Rust performance contingent on a C malloc? Because Rust switched to “system” allocators way back for compatibility with, well, the system, as well as introspection / perf tooling, to lower the size of basic programs, and to lower maintenance. It used to use jemalloc, but that took a lot of space in even the most basic binary and because jemalloc is not available everywhere it still had to deal with system allocators anyway.
- flohofwoe 1y agoSo basically, the Rust project made a bad decision and now it's all musl's fault? ;)
- buster 1y agoSounds like a sane decision to me? Using musl is the developers decision, not Rusts.
- flohofwoe 1y agoIt's not a developer decision on Alpine where musl is the system allocator. Otherwise I fully agree, application developers are mainly responsible for the performance of their applications.
- rurban 1y agoRich replaced the default musl malloc some time ago for exactly those reasons. Maybe they still used the old musl libc? The new one was drafted here: https://github.com/richfelker/mallocng-draft https://github.com/richfelker/mallocng-draft
- Aissen 1y agoThis is addressed in the article: https://nickb.dev/blog/default-musl-allocator-considered-harmful-to-performance/#musls-new-mallocng https://nickb.dev/blog/default-musl-allocator-considered-har...
- masklinn 1y agoThe new allocator does nothing to improve the performances in a threaded / contended application: https://www.openwall.com/lists/musl/2025/09/04/3 https://www.openwall.com/lists/musl/2025/09/04/3
- typpilol 1y agoThe response to the link here is really telling. Blames it all on app code like Wayland
- alberth 1y agoFrom the article: > “the new ng allocator in MUSL doesn’t make a dime of a difference”
- rurban 1y agoYes, sorry, missed that at the very end.
- jauntywundrkind 1y agoAlas this is a huge foot gun that ensnares many orgs. Because engineers seem drawn like moths to the flame to Alpine container images. Yes they are small, but the ramifications of Alpine & using musl are significant. Optimizing for size & stdlib code simplicity is probably not the best fit for your application server! Container size has always struck me as such a Goodhart's Law issue (and worse, already a bad measure as it measures only a very brief part of the software lifecycle). Goodhart's Law: > When a measure becomes a target, it ceases to be a good measure This particular musl/Alpine footgun can be worked around. It's not particularly hard to install and use another allocator on Alpine or anywhere really. Ruby folks in particular seem to have a lot of lore around jemalloc, with various versions preferences and MALLOC_CONFIGs on top of that. But in general I continue to feel like Alpine base images bring in quite an X factor, even if you knowingly adjust the allocator: the prevalence of Alpine in container images feels unfortunate & eccentric. Going distorless is always an option. A little too radical for my tastes though usually. I think of musl+busybox+ipkg as the distinguishing aspects of Alpine, so on that basis I'm excited to see the recent huge strides by uutil, the rust rewrite of gnu coreutils focused on compatibility. While offering a BusyBox-like all-in-one binary convenience! It should make a nice compact coreutils for containers! The recent 0.2 has competitive performance which is awesome to see. https://www.phoronix.com/news/Rust-Coreutils-0.2 https://www.phoronix.com/news/Rust-Coreutils-0.2
- jurschreuder 1y agoI'm not a fan of Rust I'm more of a C++ guy but Ripgrep is also nice I always install it.
- Karrot_Kream 1y agoHuh I guess I'm lucky I never faced this, we've always used Debian or RHEL containers where I've worked. Every time I toyed with using a minimalist distro I found debugging to be much more difficult and ended up abandoning the idea. Once the container OS forks and runs your binary, I'm curious why does it matter? Is it because people run interpreted code (like Python or Node) and use runtimes that link musl libc? If you deploy JVM or Go apps this will probably not be a factor.
- znpy 1y agoAnother day, another reason to avoid musl libc
- time4tea 1y agoMaybe its just that the allocator is absolutely fine for single thread programs, and that's what a lot of programs are... Its not so long ago that the GNU libc had a very similar allocator too, and thats why you'd pop Hoard in your LD_PRELOAD or whatever. Not every program is multi-threaded, and so not every program would experience thread contention.
- simonask 1y agoHot take: Almost all programs are actually multithreaded. The only exception is tiny UNIX-like shell utilities that are meant to run in parallel with other processes, and toy programs. The third exception is programs that should be multithreaded but aren't because they are written in languages where adding more threads is disproportionately hard (C, C++) or impossible (Python, Ruby, etc.).
- sim7c00 1y agohow are C/C++ disproportionally hard? the concept of multi-threading is the same for any language that supports it, most of the primitives are the same, and it's really not a lot nor complicated code to implement those. the difficulty totally lies in the design... actually using parallelism where it matters. - tons of multi-threaded programs are just single-thread with a lot of 'scheduler' spliced into this one thread -_-
- nrdvana 1y agoMaybe the large number of standard library functions that operate on globals and require you to remember the "_r" variant of that function exists, or the mess with handling signals, or the fact that Win32 and Posix use significantly different primitives for synchronization? Or maybe just the fact that most libraries for C/++ won't have built-in threading support and you need to synchronize at each call site? Unless I'm writing Java, I avoid multithreading whenever possible. I hear it's also nice in Go.
- simonask 1y ago
- fyrn_ 1y agoThe musl pthread muxtexes are also awfully slow: https://justine.lol/mutex/ https://justine.lol/mutex/
- deleted 1y ago[deleted]
- torginus 1y agoThe root cause of the issue, is that musl malloc uses a single head, and relies on locking to support multiple heaps. This means each allocation/free must acquire this lock. Imo it's good for single threaded programs (which might've been musls main usecase), but Rust programs nowadays mostly use multiple threads. In contrast mimalloc, a similarly minimalistic allocator has a per-thread heap, which each thread owning the memory it allocates, and cross-thread free's are handled in a deferred manner. This works very well with Rust's ownership system, where objects rarely move between threads. Internally, both allocators use size-class based allocation, into predefined chunks, with the key difference being that musl uses bitmaps and mimalloc uses free lists to keep track of memory. Musl could be fixed, it they switch from a single thread model, to a per-thread heap as well.
- flohofwoe 1y ago> a similarly minimalistic allocator mimalloc has about 10kloc, while (assuming I'm looking in the right place) the new musl allocator has 891 and the old musl allocator has 518 lines of code. I wouldn't call an order of magnitude difference in line count 'similar'.
- torginus 1y agoIt's minimalistic in the sense that it compiles to a tiny binary (a lot of the code is either per platform, musl is POSIX only afaik) or for debugging. Yes it's bigger, but still tiny compared to something like jemalloc, and I'm sure it's like 10kb in a binary.
- adgjlsfhk1 1y agoyeah, the Mimalloc design is just the correct one.
- anthk 1y agoChimera Linux did some changes on their distro because of that. EDIT: Ah, they were mentioned, of course. On some malloc replacements, telescope -a gopher/gemini client- used to be a bit crashy until I used jemalloc on some platforms (with LD_PRELOAD). Also, the performance rendering pages with tons of links improved a lot.
- flarecoder 1y agoFor docker images, cgr.dev/chainguard/wolfi-base (https://images.chainguard.dev/directory/image/wolfi-base/versions https://images.chainguard.dev/directory/image/wolfi-base/ver...) is a great replacement for Alpine. Wolfi is glibc based. It's easy to switch from Alpine since Wolfi uses apk for package management with similar package names and also contains busybox like Alpine.
- dijit 1y agoI’d much rather go with distroless, if its a choice. But I think you can tweak musl to perform well, and musl is closer to the spec than glibc so I would rather use it; even if its slower in the default case for multithreaded programmes.
- masklinn 1y ago> But I think you can tweak musl to perform well You can not, its allocator does thread safety via a big lock and that’s that. > musl is closer to the spec than glibc Is it? > even if its slower in the default case for multithreaded programmes. That’s far from the only situation where it’s slower though.
- dijit 1y agoYeah, the musl people tend to closely follow the spec, this doesn’t always win them friends: https://news.ycombinator.com/item?id=22682510 https://news.ycombinator.com/item?id=22682510 Swapping out jemalloc for the system allocator will net you huge performance wins if you link against musl, but you’ll still have issues with multithreading performance due to the slower implementations of necessary helpers.
- adgjlsfhk1 1y agoSometimes the spec sucks. A lot of the UNIX specs were written before anyone knew how to program multi-threaded systems, and thus are impossible to implement correctly (setenv is probably the most famous example)
- flohofwoe 1y agoMy simple rule of thumb: if the general purpose allocator shows up in performance profiles, then there's too much allocation going on in the hot path (e.g. depending on the 'system allocator' being fast in all situations is a convenient but sloppy attitude for code that's supposed to be portable since neither the C standard nor POSIX say anything performance).
- saagarjha 1y agoThey don't, but if your C standard library is slow you should get a new one.
- flohofwoe 1y agoFWIW on Emscripten I specifically pick the slow-but-small emmalloc instead of the fast-but-big jemalloc because a small size matters more than performance in that case. My C code also rarely heap-allocates, and the few heap-allocations that happen are all in the init-phase, not in the hot path - e.g. even in multithreaded code, the MUSL allocator would be totally fine. Performance in edge-cases by far isn't the only metric that matters for allocators.
- w15v 1y ago[dead]
- MereInterest 1y ago> Corollary: hats off to Red Hat for supporting their distro releases for such a lengthy period of time. This has been my bane at various open source projects, because at some point somebody will say that all currently supported Linux distributions should be supported by a project. This works as a rule of thumb, except for RHEL, which has some truly ancient GCC versions provided in the "extended support" OS versions. * The oldest supported versions in "production" is RHEL 8, and in "extended support" is RHEL 7. * RHEL 8 (released 2019) provides gcc 8 (released May 2018). RHEL 7 (released 2014) provides gcc 4.8 (released March 2013). * gcc 8 supports C++17, but not C++20. gcc 4.8 supports most of C++11 (some C++ stdlib implementations weren't added until later), but doesn't support C++14. So the well-meaning cutoff of "support the compiler provided by supported major OS versions" becomes a royal pain, since it would mean avoiding useful functionality in C++17 until mid-2024 (when RHEL 7 went from "production" to "extended support") or mid-2028 (when RHEL 7 "extended support" will end). It's not as bad at the moment, since C++20 and C++23 were relatively minor changes, but C++26 is shaping up to be a pretty useful change, and that wouldn't be usable until around 2035 when RHEL 10 leaves "production". I wouldn't mind it as much if RHEL named the support something sensible. By the end of a "production" window, the OS is still absolutely suitable as a deployment platform for existing software. Unlike other "production" OS versions, though, it is no longer reasonable as a target for new development at that point.
- phoronixrly 1y ago> at some point somebody will say that all currently supported Linux distributions should be supported by a project Ask for payment for extended support as well.
- Aissen 1y agoRHEL has gcc-toolset-N (previously devtoolset-N-gcc) for that. It's perfectly fine to only support building a project with, say, the penultimate gcc-toolset. Or ask for a payment for support, which is the norm in this (LTS) space.
- MereInterest 1y agoOh, absolutely, and I usually push for having users installed a more recent compiler. The problem comes when the compatibility policy is defined in terms of the default compiler provided, because then it requires a larger discussion around that entire policy.
- kchoudhu 1y agoCan someone please write a '"considered harmful" considered harmful' piece.
- mtndew4brkfst 1y agoHere's one from 2002: https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html
- kchoudhu 1y agoPerfect.
- M95D 1y agoQuote from the musl mailing list: > The mallocng allocator was designed to favor very low memory overhead, low worst-case fragmentation cost, and strong hardening over performance. This is because it's much easier and safer to opt in to using a performance-oriented allocator for the few applications that are doing ridiculous things with malloc to make it a performance bottleneck than to opt out of trading safety for performance in every basic system utility that doesn't hammer malloc. [1]https://www.openwall.com/lists/musl/2025/09/05/3 https://www.openwall.com/lists/musl/2025/09/05/3
- valyala 1y agoWe also hit this scalability issue at the default memory allocator for musl. Switching back to glibc allowed increasing the performance in production by 5x on a machine with 96 CPU cores [1]. [1] https://github.com/VictoriaMetrics/VictoriaLogs/issues/517 https://github.com/VictoriaMetrics/VictoriaLogs/issues/517