5 ms·
Don't use musl if you care about performance
- grep_it 1mo agoI think the size of linked binaries and simplicity were always the main features?
- Joker_vD 1mo agoYeah... I've recently had a chance to compare how fgets is implemented in both GNU libc and musl, and, well. With glibc, it was a challenge to even find where the fgets's code actually is.
- jdc-pub 1mo agoI haven’t dug into why, but for unknown-linux builds on x86, Rust binaries have been substantially smaller on musl than standard dynamic linking to glibc, for me. No idea if I’m doing something wrong or if the handful of cases I tried were all special in some way.
- marssaxman 1mo agoPeople have such different perspectives. 26% slower does not sound "terrible" to me; it sounds like quite a reasonable price one might choose to pay for the convenience musl offers. If musl's allocator were 2.6x slower, I might call that "not so great"... but in order to qualify as "terrible" I think the difference would have to be an order of magnitude!
- loeg 1mo agoThe 26% number at the top of the article is from using mimalloc (which is a high performance allocator, at least as fast as the glibc allocator) + musl for some task, and the slowdown is coming from (probably) slow musl implementations of memcpy/memset. The musl allocator is even worse.
- Joker_vD 1mo ago> the slowdown is coming from (probably) slow musl implementations of memcpy/memset. It's wild that such a fundamental piece of code (you can't really implement operation on structs without those) is library-supplied. I wish compilers would just have something like __builtin_memcpy and __builtin_memset, and provided some highly optimized, specialist-crafted assembly in those, instead of having to inline the library code and hopefully be able to optimize it.
- kvuj 1mo agoMaybe you're being sarcastic, but I'm pretty sure clang + gcc do offer these. The problems at first glance : - Not having control over the implementation detail of the interface that your library provides is probably not wise. Sounds like a lot of bad bug reports and edge cases that you have no control over. - Not all compilers may provide these.
- wahern 1mo agoAs others have alluded, __builtin_memcpy doesn't resolve to a runtime implementation. GCC and clang treat functions like memcpy specially. Because they're defined by the standard and are reserved names, compilers can assume the exact semantics specified by the standard and elide library calls altogether with optimized inline code. But if the compiler can't do the optimization (can't prove alignment, indeterminate length, etc), it just emits a library call, even if your source has some other local function definition named "memcpy". Explicit use of __builtin_memcpy is treated identically to calls to memcpy, unless the compiler is invoked with -ffreestanding, in which case it only optimizes __builtin_memcpy and skips special treatment of calls to memcpy, but __builtin_memcpy could still expand into a call to memcpy. If you're writing a C library you want to use -ffreestanding. (I think. There may be more nuance. More info at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56888 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56888)
- compiler-guy 1mo agoClang and GCC do provide these, and automatically use them in many situations (particularly small copies). But c-libraries can actually do it better in many cases, especially for large copies. Glibc, for example, has perhaps ten different implementations of memcpy just for x86. The compiler certainly could provide all that, but the next step is harder: glibc automatically dispatches to the proper one at runtime based on the actual microarchitecture that the binary is running on. You pay the extra dispatch cost once, but all of non-inline function call cost every time. This is what allows distros to compile to a nice baseline architecture, but still get near-optimal memcpy performance on many more architectures than a single inline instance could possibly give. These differences matter. And it does it for not just memcpy, but half-a-dozen other extremely performance sensitive library functions, like strcpy and so on. Inlining works very much against this strategy. If you can guarantee that the target microarch never changes, then it isn't a good one. But that is somewhat unusual for everyone but those who build their own binaries to run on a single class of machines forever. Worse, inlining the really high performance versions of these ends up being terrible from a code size perspective, because they are often hundreds of instructions, which can have bad caching effects. And once you amortize the function-call cost over many iterations of the loop, it isn't so expensive to call out to the library. Anyway, just some additional considerations to think about.
- stackskipton 1mo agoOps here, I think if you NEED that convenience, sure, rock with MUSL BUT I also see a ton of devs crowing about using MUSL on my 128GB x86 Kubernetes hosts. I have plenty of Disk Space, you can ship glibc based container.
- deleted 1mo ago[deleted]
- SkiFire13 1mo agoThe 26% slower appears to be for their whole application, not just the allocator. For some parts of the application to make the whole this much slower it must mean that those parts are quite a lot slower, likely much more than 2x. Moreover the 26% is with mimalloc, with musl's allocator it's 144%, so there are likely other parts that are slower (likely the memcpy implementation)
- fhn 1mo agoTell your employer a 26% pay decrease for you is acceptable.
- otterley 1mo agoI'm curious. What convenience, specifically, are people benefiting from by using musl?
- sombragris 1mo agoThe convenience (?) of not having to comply with GPL terms. Some people really hate copyleft.
- ApolloFortyNine 1mo agoGlibc is lgpl, you _definitely_ don't need to comply with GPL to link to it.
- sombragris 1mo agoRight. I should have said "comply with copyleft licenses".
- plorkyeran 1mo agoIf you want to ship a prebuilt binary that'll run on any linux distro you need to statically link libc and musl is by far the easiest way to do that.
- jcelerier 1mo agoexcept it only can work for CLI apps or anything that doesn't use the GPU as for instance nvidia drivers require glibc
- wakawaka28 1mo ago26% slower could turn into a huge hardware bill, and could render the library unusable for some purposes. There are many applications for which 26% is negligible, but it ain't nothing...
- jcelerier 1mo ago> for the convenience musl offers ... you're fine with trading an app running 26% slower, to save a dozen megabytes ? that sounds positively insane to me. That's accepting to go from e.g. 60fps to ~45 fps (e.g. completely unacceptable)
- atiedebee 1mo agoThat would depend on the application. The convenience isn't just a smaller binary, but also being independent of the distros glibc version (being on an LTS distro and not updating often, I have experienced incompatible glibc versions often enough). The megabytes shaved off are significant if the final binary size is < 1MB, which is a completely different size class. And not all applications are performance sensitive. Something like UNIX's bc command would benefit more from having easier compatibility and a faster startup than more optimized allocators and string functions. In the end, it ~doesnt even matter~ is all trade-offs. The nice thing is that it is up to the developer to make the decision of which libc to use, so everyone gets what they want.
- mococa 1mo agoAlso musl is not a complete runtime
- OptionOfT 1mo agoAre there other options if I want to ship a 'FROM scratch' image with just a single Rust executable, and everything compiled in? That to me is the main driver for MUSL.
- nazgulsenpai 1mo agoSame usecase here, I use musl for compiling self contained Nim utilities I use on containers and servers without having to deal with glibc hell.
- masklinn 1mo agoThe best option would be the x86_64-unknown-linux-none target (https://doc.rust-lang.org/nightly/rustc/platform-support/x86_64-unknown-linux-none.html https://doc.rust-lang.org/nightly/rustc/platform-support/x86...) however it currently a tier 3 with a single maintainer (so it is technically available but unsupported). If this specific use case is of high interest to you and you have some available bandwidth, contributing to it, maybe becoming a maintainer, and eventually organising a tier 2 MCP would definitely be a good idea.
- VorpalWay 1mo agoNote that this is no-std no-alloc target, with all the limitations that leads to. You could add alloc with a custom global allocator, but I don't even know what high perf global allocator you could use that wouldn't need libc. Jemalloc and mimalloc are out. Some embedded allocators would work (but those are rarely high performance, instead being optimised for small code and data footprints). That said, with enough effort (quite a lot!) it would be possible to add support for alloc and std without libc on Linux specifically (since it has a stable syscall ABI). What might be more realistic though is looking at relibc (a rust implementation of libc, made for Redox OS but from what I read it also supports Linux). But I haven't tried it and I don't know the state (or goal) of it.
- masklinn 1mo ago> it would be possible to add support for alloc and std without libc on Linux specifically (since it has a stable syscall ABI). Well yes that’s a Linux specific target so that’s kinda the point. Technically you could do libcless on a few other platforms which are not actively hostile to it (yet) like freebsd, but that would have no chance of getting to tier 2 if it was even accepted.
- desdenova 1mo agoMost of musl's performance issues come from their allocator. Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss.
- loeg 1mo ago> Most of musl's performance issues come from their allocator. Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss. This is addressed and disputed very early in the article. The very first benchmark presented shows a 26% regression using musl + mimalloc, a high-performance 3rd party allocator.
- masklinn 1mo agoIt's not really disputed since musl without mimalloc has a 144% overhead, so most of the performance issues do indeed come from the allocator, by a pretty large margin (~85% of it). Not only that, but some of the "other code" performance hit might still come from the allocator: when you set a global allocator on the Rust side, musl still uses its own allocator internally (as demonstrated by https://github.com/BurntSushi/ripgrep/issues/3494 https://github.com/BurntSushi/ripgrep/issues/3494). And the compounding issue is that the allocator issues get significantly worse as parallelism increases, as the allocator is serial, so as concurrency increases so does the impact of the allocator, which is not the case for most of the "regular slow" code (of musl), those have a relatively constant overhead per thread.
- loeg 1mo ago"Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss" is disputed; 26% is not "very little," even if 144% is worse.
- bloppe 1mo agoThis is only one aspect of "performance". glibc's allocator may be faster, but it also uses more memory. For a much more technical discussion, see https://github.com/sharkdp/fd/issues/710 https://github.com/sharkdp/fd/issues/710
- legastenigga 1mo ago[dead]
- mattrighetti 1mo agoPosted this the other day but the whole musl allocator thing seems to be well known [0] [0]: https://news.ycombinator.com/item?id=45143347 https://news.ycombinator.com/item?id=45143347
- masklinn 1mo agoMusl's allocator being awful is pretty well known, though mostly in that it's absolutely awful in multithreaded context. TFA points out that musl has a bunch of other noticeably slower functions, which is less well known (though they're also slower by a smaller factor, and they don't worsen as your parallelism increases).
- up2isomorphism 1mo agoFunny thing is that the major reason most people use musl is because glibc make it (artificially) hard to do completely static linking.
- matherial 1mo agoIf your algorithm does a ton of small allocations to the point where the allocator is the bottleneck, you're already doing it wrong. The allocator necessarily comes with a lot of overhead because it needs to accommodate diverse use cases, avoid fragmentation, and ideally, implement a variety of security checks. If you're doing something alloc-intensive, you're probably allocating and freeing a lot of identical structures and you'd be better off grabbing some continuous memory and managing that yourself in a task-specific way. But the reality is that almost no one actually cares about performance because compute is cheaper than expertise and labor, at least in the short haul. Everything is getting more bloated and slower and we just compensate by adding CPU cores, gigabytes and gigahertz.
- haolez 1mo agoThat's an interesting viewpoint, but then, will the allocator's performance never matter for any use case that is not "wrong"? It doesn't feel right.
- gumby 1mo agoThink about it this way because the issue isn’t specific to allocators: it’s pretty good in general but can often be beaten if you have special understanding of what you need to do. That’s OK. You can buy cars and trucks that are optimized for driving on freeways and residential streets carrying stuff people often carry. But then there are special vehicles like fork lifts and such that are kinds of large special cases. And then there are weirdo specialised vehicles that have four wheels but are rare and their users can’t live without them. Languages like C++ let you plug in special allocators if you want. But most people don’t. Some, like HFT people do crazy headstands to avoid slow allocations. I don’t ever want to do that but if they want to, why not. I don’t think they complain that the default case doesn’t fit their needs!
- matherial 1mo agoFor a typical program, I bet that the overall impact of the glibc allocator is well under 0.1%. If you can choose between < 0.1% and < 0.12%, I guess it matters in some sense, but not in any practical way. We almost never spend time on other sub-0.1% optimizations. You could probably squeeze a lot more by optimizing CPU branch predictor performance, minimizing CPU cache misses, or fine-tuning the scheduling strategy, but we also don't bother. 'Tis is the era of "native" apps written in Electron.
- dchest 1mo ago[dead]
- tombert 1mo agoI feel with Rust I try and avoid re-allocations in most cases anyway, so I'm not sure that musl's allocator being slow would significantly affect performance (though I haven't benchmarked it). I feel like part of the appeal of Rust is that you can do imperatively-style mutation-heavy code comparatively risk-free, so despite me normally being the "Functional Programming Nerd", I generally write Rust in a style that's a bit closer to C. I use musl for my Rust stuff because I have noticed that for the stuff I write it appears to have a lower memory footprint; since a lot of what I do is IO-bound anyway, I care more about using less memory than raw performance.
- superdisk 1mo agoI swear "bifrost" has to be the most overused name in computing, possibly only behind "yggdrasil." I'm not sure what's so magnetic about those names but I've seen at least 10 different things called that.
- lrvick 1mo agoI really find complaints about the minimum viable placeholder malloc in musl confusing. Does anyone seriously try to use malloc-ng for performance critical use cases? Our entire linux distro is musl based BUT we swap out the default malloc with mimalloc for high performance because why would you not? Best of both worlds. https://codeberg.org/stagex/stagex/src/branch/main/packages/core/musl/Containerfile#L54 https://codeberg.org/stagex/stagex/src/branch/main/packages/...
- madduci 1mo agogood article points, but what's a good alternative Docker Image with a small footprint and a package manager to Alpine Linux, if you need to install something on it and powered by glibc?
- rdsubhas 1mo agoOps here. Don't use musl if you care about disk size or compatibility either. For reference: docker.io/node:26-alpine - 61MB docker.io/node:26-trixie-slim: 84MB gcr.io/distroless/nodejs26-debian13: 55MB Before this post, we have issues with different, slower DNS resolution in musl. In 2026, there in no reason other than self-inflicted compatibility and performance pain to use musl in production services. It has it's place, for binary cross-OS distribution, and generally having an alternate C STL. But avoid for SaaS.