9 ms·
Porting systemd to musl Libc-powered Linux
- bhaney 2y ago> boots in 1/3rd the time of OpenRC on the same system Wow, that's a suspiciously impressive difference. I was under the impression that OpenRC and systemd support similar parallel service startup features and usually boot systems in approximately the same amount of time, with maybe a slight edge to systemd. 3x speed makes me wonder if there's something else going on, like the current early systemd port having some bugs that cause it to incorrectly skip some important work. Either way, super cool. What's the likelihood that this ends up getting upstreamed?
- frankjr 2y ago> I was under the impression that OpenRC and systemd support similar parallel service startup features It's not enabled by default in OpenRC because it's not stable. > WARNING: whilst we have improved parallel, it can still potentially lock the boot process. Don't file bugs about this unless you can supply patches that fix it without breaking other things! https://github.com/OpenRC/openrc/blob/ea310b2e580a25038f9592da4160d017ebfce3fc/etc/rc.conf#L7 https://github.com/OpenRC/openrc/blob/ea310b2e580a25038f9592...
- NavinF 2y agoI'm amazed projects like that are still active. This isn't a case where systemd is ahead of the curve, every other OS also starts services in parallel.
- amszmidt 2y agoPlenty don’t, e.g. BSDs. Bunch of embedded GNU/Linux systems. Etc.
- NavinF 2y agoI meant OSs that have consumer market share. Eg on people's desktops and phones. Nobody cares about BSDs and a bunch of embedded GNU/Linux systems
- awilfox 2y agoA lot of people care about BSDs and embedded Linux systems. Such as: Netflix (FreeBSD), anyone with a Linux-based router or cable/DSL modem, and so on. That said, parallel startup is possible on those systems too. There was an out-of-tree patch for parallel rc(8) on FreeBSD some time ago, and dinit is parallel, etc. I think OpenRC should move towards making parallel startup stable and supported so that those systems can do that as well.
- zekica 2y agoAlmost everyone's router is an embedded Busybox/Musl/Linux system. There are billions of them running every day.
- synergy20 2y agothey don't use openrc though
- btreecat 2y agoAlpine has market share. It's just targeting a different market space than what you're measuring.
- deleted 2y ago
- bee_rider 2y agoIt is a simple little startup system with some dedicated users. Open source projects don’t need huge user-counts to justify their existence. It isn’t a profit center, people just do this stuff for fun, because they enjoy tinkering, or maybe they coded it up and decided to just share it out of altruism.
- happymellon 2y agoFolks who evangelise these "just as good as systemd" solutions always hide a key factor. There is a reason systemd took over, its massively supported and widely tested.
- actionfromafar 2y agoIt couldn't be massively supported until it took over, though? It's more than a little platform effect going on, it's not all by virtue of systemd being just all over great. It has some great ideas but it's an entire world unto itself which feels very NIH-ish. Edit: I think it serves as both a warning and an inspiration. Don't let a duct tape solution (such as for instance boot via rc shell files) live for too long, or it might be replaced by something baroque.
- trueismywork 2y agoIdeas are just 1% if implementation. Systemd (while being a little problematic in beginning) has had huge amount of amazing work being done in it.
- p_l 2y agoDoesn't mean the result is all that good. Having dealt with it for some time now, the amount of "good idea, bad implementation" I hit is too damn high. And it's nigh impossible to replace for various reasons, including at one point critical interfaces that various user-important programs depended on being a constant treadmill of updates, lack of documentation, and hidden dependencies on internals.
- awilfox 2y agoIt's true. systemd has a lot of good ideas with questionable implementations. I'd love networkd if it didn't have silly decisions made. And it has some questionable ideas with good implementations. For example, I hate the journal, but admit it is implemented well. At the end of the day, it's about tradeoffs and using what suits the system at hand. For some people that will be systemd, and bringing systemd to Adélie and other musl libc distributions means that it can be used by those people. Call me old-fashioned, but I also hold out hope that at least some of those questionable implementations could be fixed if only someone with the desire would write a patch and send it upstream. Bringing systemd to musl means the people in musl land that aren't knee-jerk anti-systemd maybe might be more enthused to do just that, improving it for everyone using it.
- awilfox 2y ago> Either way, super cool. What's the likelihood that this ends up getting upstreamed? High: <https://github.com/systemd/systemd/pulls?q=is%3Apr+author%3Aawilfox https://github.com/systemd/systemd/pulls?q=is%3Apr+author%3A...>.
- blueflow 2y agoDid Lennart change his mind? https://lists.freedesktop.org/archives/systemd-devel/2014-October/023869.html https://lists.freedesktop.org/archives/systemd-devel/2014-Oc...
- kokada 2y agoI don't think Lennart changed his opinion, if you look further in this thread[1] Lenart wasn't ever against musl per se, it is more like "we use non-POSIX features from glibc that we found useful, and we are not going to reinvent them, so either convince musl to implement those features or carry patches fixing the issues". That I kind concur with him, instead of workaround missing functions it may be better to just write a compatibility library that complements musl with the missing functions. This way, any program that needs those functions could just import this library if they're compiling in a non-glibc system. I am seeing that lots of the fixes proposed by the author are things like "missing import because glibc exposes some symbols by mistake and musl does not". And this is definitely the kind of fix that upstream would accept[2]. [1]: https://github.com/systemd/systemd/issues/10130 https://github.com/systemd/systemd/issues/10130 [2]: https://github.com/systemd/systemd/pull/34066 https://github.com/systemd/systemd/pull/34066
- actionfromafar 2y agoIt must be something in the air. I find flatpak, appimage, snap etc underwhelming. The go reimplementation¹ of appimage looks promising, but it would be nicer if one could more easily link Linux applications statically. Another idea would be to build a "meta-libc", MIT licensed, which exports a glibc compatible interface but called musl or whatever else libc under the hood for implementation. This is slightly complicated by the fact that musl does not export any version define or symbol. Or just fork musl and start adding glibc-isms to it... 1: https://github.com/probonopd/go-appimage https://github.com/probonopd/go-appimage
- Y_Y 2y agoI'm with you, if what you're saying is that containerized applications are a cop-out. On the other hand, they have their uses. For example Valve seems to have a hard time packaging Steam (being a weird 32bit web browser with DRM), but the flatpak (produced independently) works nicely, and because it's closed source it's your best option. (Also I think it's too kind to include Snap with the others, I find it uniquely objectionable since it's neither convenient nor functional.) edit: clarify that the Steam flatpak is not made by Valve
- ELFfBtiN1yLzonf 2y ago> their flatpak Steam flatpak was not created by and is not supported by Valve. It's an unofficial project, and flathub says as much (I'm assuming you mean the flathub package): https://flathub.org/apps/com.valvesoftware.Steam https://flathub.org/apps/com.valvesoftware.Steam > Unverified https://github.com/flathub/com.valvesoftware.Steam https://github.com/flathub/com.valvesoftware.Steam
- ELFfBtiN1yLzonf 2y agoA user-friendly wrapper around nix would be much more functional (in all meanings of the word) and interesting than any of those solutions. Unfortunately, the world is moving in the opposite direction.
- 2y ago
- dmwilcox 2y agoit's funny that many people mention boot time as a metric when for most servers, VMs and laptops the devices are either always on or sleeping. startup times do nothing in those cases. personally I use a custom alpine live-usb that is immutable and I fully shutdown and start-up all the time. openrc doesn't give me any grief, the bulk of my startup time is copying my 500mb rootfs into a tmpfs systemd for me is solidly "at work" software, and not stuff I'm a big fan of either (journald, networkd and the silly dns resolver all have serious issues). so please, keep away from my alpine or I'll have to fork -again- :)
- zja 2y agoBoot times are pretty important for serverless environments. Alpine, which uses musl, is very popular in those situations.
- blueflow 2y agoI thought serverless environments are containers and thus don't need to boot?
- amaccuish 2y agoThey still have a userspace to bring up.
- djbusby 2y agoAnd don't need systemd for that.
- tuetuopay 2y agonowadays serverless environments are microvms. that is, a standard machine type that is explicitly virtual (so device drivers have no wait time for hw to init). however, they don't either need userspace to boot because the function/application/whatever is likely init itself or not far off. so they need to boot, but only the kernel needs to boot.
- 2y ago
- ducktective 2y agoHas there ever been an empirical study on the performance hit of switching from glibc to musl?
- IshKebab 2y agoThese ones found no difference: http://www.etalabs.net/compare_libcs.html http://www.etalabs.net/compare_libcs.html https://users.rust-lang.org/t/optimizing-rust-binaries-observation-of-musl-versus-glibc-and-jemalloc-versus-system-alloc/8499 https://users.rust-lang.org/t/optimizing-rust-binaries-obser... This guy found Musl much slower for multithreaded allocation: https://www.linkedin.com/pulse/testing-alternative-c-memory-allocators-pt-2-musl-mystery-gomes/ https://www.linkedin.com/pulse/testing-alternative-c-memory-... These found Musl a bit slower with LTO: https://github.com/vectordotdev/vector/issues/2313 https://github.com/vectordotdev/vector/issues/2313 Except for that LinkedIn one (which feels like it might be a bug), it seems like there isn't really much in it, which is what I'd expect tbh. Kind of like Clang vs GCC. Sometimes one is faster, but probably not by much.
- SubjectToChange 2y agoNot that I'm aware of. However, I would expect large regressions (20%+) in general, e.g. the malloc implementation in musl is famously primitive.
- jeffbee 2y agoHow could anyone who cares about allocator performance be satisfied with glibc's allocator in the first place?
- SubjectToChange 2y agoYeah, I suppose that's fair. Still, the musl malloc implementation is just the most infamous example of its subpar performance. Obviously glibc comes with an enormous amount of functionality and performance optimizations. Somewhat unrelated, but I'm looking forward to llvm-libc becoming a practical replacement for musl. I'd love a modern, non-gnu libc implementation that isn't so dogmatic.
- raverbashing 2y agoLooking at some patches it seems some of those might just be bugfixes that are inconsequential (or maybe just undetected) on libc?
- synergy20 2y agoboth systemd and musl developers were talking here: https://github.com/systemd/systemd/issues/10130 https://github.com/systemd/systemd/issues/10130 it seemed like a deadlock
- ruthmarx 2y agoHonestly thank goodness. Choice is great and all that but I don't want system to infect Alpine or it's ecosystem at all. I never, ever want to see an apk that requires systemd.
- bigstrat2003 2y agoSo... choice is great, except when it comes to something you personally dislike?
- yjftsjthsd-h 2y agoNo; choice is great, and sometimes the only way to preserve one choice is to limit another.
- consteval 2y agoI don't know where this widespread hallucination that systemd is "forced" came from. You're allowed to use another init system, and the distros CHOSE systemd. Nothing is stopping you, it's just that nobody does it because: 1. Pretty much nothing is as good as systemd as a whole project. Stringing together 30 random packages is a huge pain. 2. Other init systems are much slower, and turns out people care about performance. 3. Other init systems are less functional, with less features which some users want. 4. Writing a competing init system is very hard, while complaining is both easy and fun. In the hypothetical case of Alpine switching to systemd, you're 100% free to maintain your own Alpine distro or even maintain your own init system or just switch to a different one.
- ruthmarx 2y ago
- RVuRnvbM2e 2y agoIf alpine gets systemd it would be an amazing bare metal server or workstation distro. Openrc lacks so many features by comparison.
- djbusby 2y agoAs a user of both, I don't really see anything missing from OpenRC. Maybe parallel startup? Hasn't been a bottleneck for me, ever. I prefer the "syslog" way over journald but that's not anything like a deal breaker. What's features I'm missing?
- boudin 2y agoIt might have changed since I last used openrc but I remember there was no user services (I find this quite useful with systemd) and there was no socket activated services neither which is quite nice.
- yjftsjthsd-h 2y ago> It might have changed since I last used openrc but I remember there was no user services Was looking into this exact thing this week. Amusingly, the suggestion I read on the gentoo wiki is to run a user instance of runit, which... I actually like that, but it does start to beg the question of why not use that everywhere.
- WhyNotHugo 2y agoSomeone is actively working on user services, but it's not merged yet: https://github.com/OpenRC/openrc/pull/723 https://github.com/OpenRC/openrc/pull/723
- RVuRnvbM2e 2y agoAll the declarative security sandboxing stuff. Last I checked openrc doesn't let you easily run a service in a process or network namespace. Or filter syscalls etc. I also make heavy use of user services, TPM encrypted secrets, systemd tmpfile directories, etc. Another thing: systemd-networkd is the only Linux networking stack I've managed to get working reliably with tailscale's dns and a private dns service on my workstation.
- synergy20 2y agoPopular linux based embedded project called yocto will benefit a lot when systemd and musl works well together, using systemd-glibc it brings a few MegaBytes to the build, which is sometimes too large. I'm sure musl can bring down the size a lot.
- ghostpepper 2y agoThis was a really well written blog post. I felt like I was able to follow the author's debugging journey very clearly. Separately I've not heard of Adelie Linux and from a quick glance at their site I can't really tell what niche they're trying to fill. I gather they use musl and (before this post) openRC, and they aim for POSIX compatibility for legacy systems.
- M95D 2y agoI recently removed most of openrc and started using only a custom inittab. The boot time was shortened by at least 10s and I have more control over what starts and when. I'm going to absolutely hate the day when firefox will depend on systemd (or systemd replaces firefox, whatever happens first).
- M95D 2y ago[flagged]
- dezgeg 2y agoIt's already been in the mid range (~1 GHz CPU, ~1 GB RAM) embedded systems over half a decade.
- consteval 2y ago> because people are too lazy to write their own init scripts Why would I bother writing an init script when an existing solution, written in C, is right there?
- M95D 2y agoWe're talking about musl systems, yes? That's routers, ip cameras, ... That C you're talking about still needs to read the config file (or files) from slow storage and then parse and apply them. Bash/Busybox ash does the exact same thing and it's also written in C. In fact, the fastest, smallest boot process is busybox init + inittab + busybox again (ip, udhcpc, klogd, sysctl, mdev, mount, etc.). There are only 2 files to read from storage: bb and inittab. But then again, dev don't care about efficiency. They only care about how easy it is for them to write the stuff. So, systemd ftw!!!
- consteval 2y ago> Bash/Busybox ash does the exact same thing and it's also written in C Yes, with the added benefit of an interpreter and shell implementation in between. Yippee! > dev don't care about efficiency systemd IS efficient. The hard truth is that it outperforms every other init system and it's not even close. You could argue "well you don't need a fancy shmancy init system at all!!" - but even embedded applications grow and new requirements are added. Then what? Your "efficient" solution sucks, and isn't fast. Whoops!
- stuaxo 2y agoThis is great, not just for users of systemd but for systemd itself, and the projects around it getting improved.
- piuantiderp 2y agoWhy even use musl if you're going to use systemd?
- e12e 2y ago> And that is how I became the first person alive to see systemd passing its entire test suite on a big-endian 64-bit PowerPC musl libc system. Wow :)