15 ms·

Why systemd is a problem for embedded Linux

208 points by synergy20 2y ago

is_taken 2y ago
You could checkout void and alpine.
userbinator 2y ago
The BSDs may be more to your liking. They're a lot more minimalist than Linux now.
- Aurornis 2y ago
  In the embedded world, BSP/driver support is the main thing that matters. Linux is undefeated in that area for embedded purposes.
  - moron4hire 2y ago
    I thought there was a compatibility layer available that made it possible to run Linux drivers on at least FreeBSD.
    - CorrectHorseBat 2y ago
      That's for userland, not drivers. Linux doesn't have a stable interface for drivers so a compatibility layer would be a nightmare to maintain.
      - moron4hire 2y ago
        That's Linuxulator, but there's also LinuxKPI: https://wiki.freebsd.org/LinuxKPI https://wiki.freebsd.org/LinuxKPI
  - bluGill 2y ago
    In embedded you should choose hardware that works. Should is key, all too often the, hardway guys don't ask they just give software something.
    - nomel 2y ago
      > In embedded you should choose hardware that works. I've never seen it approached with that kind of incompetence, in my professional life. Embedded is a system thing, not a hardware thing. You pick the system that will work. Ignoring the software side of things is ignoring the majority of the problem in embedded work. I would have agreed with you 20 years ago, but it's 2024, and why RPI is so often used for prototyping.
      - bluGill 2y ago
        Sorry, wrote the original comment on my phone. The sentiment is close enough, but generally things are not quite that bad. I've seen hardware ask, but the person they asked didn't know how to check how much memory we actually needed. Hardware also failed to tell that person that half the memory they were asking about was dedicated to the GPU and thus not available for application code. I've also seen them choose hardware that seems to have great drivers and they can point to others in our industry who are shipping with that, only to latter hit a major bug that means we can't use it without that fixed - but we are too small a customer to be able to demand they fix bugs that only affect us. RPI is often useful for prototyping because it is cheap enough, but if you are not a very large company nobody will talk to you. This is both you can't get support from Broadcom, and worse your supply management cannot get contract signed (for small companies this doesn't matter you take what you can get, but larger companies often demand better support than they can get from pi)
solarkraft 2y ago
> The more fundamental problem is that the people who most like systemd are distribution managers. (...) a distribution manager doesn’t have to maintain, integrate, and support a whole bunch of system utilities from different sources – systemd provides everything in one huge build. That’s actually a great lesson in optimizing your product for adoption.
- exe34 2y ago
  it's also why corporate software is so shit - it's shiny in all the ways it needs to be shiny for management to make the decision, but it's a pain for the actual users. the trick in both cases is to know your customers!
  - oblio 2y ago
    Turns out, many people use systemd after what, 15? years of doomsday predictions :-)
    - stepupmakeup 2y ago
      Unfortunately not by choice
    - fsckboy 2y ago
      i don't notice anything good systemd does for me, just where it interferes, and that's a constant annoyance, a few days a week.
      - viraptor 2y ago
        Can you link some upstream issues you raised about the big annoyances?
        exe34 2y ago
        not many people enjoy bikeshedding.
        2y ago
pnathan 2y ago
I can only say that the obvious answer here is to throw effort into supporting & improving Gentoo.
viraptor 2y ago
I'm conflicted on the name/issue. I don't have any issues with Linux on embedded systems, because I'd use a minimal init with that one single app it should be running. (Or as mentioned in other comment, build your minimal distro yocto-style) RPi is a low spec general computer at this point. The idea of running gnome on an embedded system just doesn't make sense in my mind... RPi took the form factor that was used for industrial control boards and made it into a usable desktop... which people use as a desktop (or home server). But it created a segment which really is neither. And a lot of the software just isn't written with that segment in mind. Maybe as it gets more popular, people will pay more attention. But currently it seems more of an expectations issue.
fyrn_ 2y ago
https://chimera-linux.org/ https://chimera-linux.org/ https://chimera-linux.org/docs/faq#what-is-the-projects-take-on-systemd https://chimera-linux.org/docs/faq#what-is-the-projects-take... Is going a really interesting direction with it's service management and login / seat management. Fwiw, I think it's author also has a more gounded perspective on systemd then many others. For Chimera the main issue is systemd's high volume use of gnulibc and gcc extensions, which is a porabiluty issue for Chimera which uses a LLVM toolchain and musl libc (and a bunch of really cool hardening and compiler engineering)
- binkHN 2y ago
  I like the direction Chimera is taking; the lead is an ex-Void maintainer and there's clearly a lot of inspiration from there.
colonwqbang 2y ago
This is anti-systemd FUD. Implying that applications which work without systemd are going to stop doing so. Suggesting that Linux distributions adopted systemd because it's good for distributors and not because users actually want it. We are "forced" to use systemd's udev implementation. Please.
- felixgallo 2y ago
  All of those things are trivially true. What are you disagreeing with?
  - viraptor 2y ago
    There are extremely few use cases where your app may want to integrate with systemd. Out of those, you get again a small percentage that can't be trivially made optional. (99.9% is just optional notification and socket passing) Outside of large projects which are interested in full system integration like gnome, nobody breaks apps to not work without systemd.
  - consteval 2y ago
    Except none of these are true, they're just things people have made up to conceal the fact they oppose systemd for ideological reasons. There's little to no technical arguments against systemd - it's fast, much more robust than what came before it, and services 99% of usecases. Users DID want systemd. Ask any sys admin, most much prefer systemd. Also systemd doesn't "force" anything. Most distributions don't even ship all systemd projects, just a couple. systemd-init IS small. The idea that systemd is a big ole monolith is just not true, and one glance at the git repo reveals that.
    - felixgallo 2y ago
      oh definitely not, and it's a matter of record. Red Hat and Poettering wanted systemd in order to control more of the platform; this is why they absorbed in udev (which had been a separate repo), came out saying that there are certain parts of the system that, despite all this being open source, were forbidden to modify or replicate, and stacked the debian voting. Systemd at the time sucked, incredibly, even worse than it does today, and routinely bricked or crashed systems. The adoption was forced and 100% political. systemd is, of course,today a continuing shambling feature factory behemoth in which hundreds of product managers try to shoehorn in more mandatory features in order to cement grip on the platform. That's why you get this ridiculous dns nonsense, this ridiculous container nonsense, this ridiculous cron nonsense, hostnamed (!), the list is endless. The technical argument is that it's a giant tasteless bag of ever increasing poorly implemented scope. The rest of Unix, by comparison, is generally not that way. If you pick up a BSD or even an alpine, you'll find that you don't need a bunch of badly written, hacked together garbage in order to get your work done. The system can be entirely readable and repeatable without cramming a bunch of cruft and poor decisions into pid 1. The idea that systemd doesn't 'force' anything is, of course, hilarious. Systemd is designed to try to be the ultimate mandatory dependency for essentially everything on the system. That's the way Red Hat wanted it to be, and that's the way it is. Every time someone mentions systemd, some random apologist dutifully trots out the idea that systemd-init is small and therefore systemd is not a monolith, checkmate. Of course, journald, libudev, localed, logind, hostnamed, homed, networkd, resolevd, systemd-boot, systemd-bsod, systemd-nspawn, timedated, timesyncd, tmpfiles, udevd, and all of the other array of dumbass bolted-on second-system-effect-driven product-managed nonsense somehow don't get mentioned.
transpute 2y ago
OpenEmbedded/Yocto, Devuan and Gentoo provide multiple init systems. systemd CVEs: https://ubuntu.com/security/cves?package=systemd&limit=100 https://ubuntu.com/security/cves?package=systemd&limit=100 Rust PoC: https://github.com/KillingSpark/rustysd https://github.com/KillingSpark/rustysd > Rustysd is a service manager that tries to replicate systemd behaviour for a subset of the configuration possibilities. It focuses on the core functionality of a service manager, not requiring to be PID1 (aka init process).. core systemd functionality is not very hard to replicate.. the advantages of having a systemd-like service manager can be brought to many other platforms.. (like alpine linux, commonly used in docker containers and small vms) and freebsd.
- dcsommer 2y ago
  How many systemd CVEs are memory or thread safety related?
  - fn-mote 2y ago
    I can't tell if this is rhetorical but... it looks like quite a few on the list. (Note: the CVE list is long but it looks inflated / full of not-serious issues.) * CVE-2022-3821 : off by one error leading to buffer overflow * CVE-2022-2526 : use after free * CVE-2021-33910 : "Memory Allocation with an Excessive Size Value" (not sure if this qualifies... not interested enough to read the source)
  - silverliver 2y ago
    What do memory and thread safety have to do with the project's stated goals? I for one would love to avoid systemd's arbitrary requirements while retaining compatibility with upstream (e.g. PID must be 1).
  - 1oooqooq 2y ago
    I'd bet you get more CVEs and actual remote exploit entry points by systemd forcing mdns, bonjour, upnp and other zero conf hacks just because that was the work the systemd team was doing in rh before.
Mave83 2y ago
For me as a user, systemd is far better than old init systems. The logging alone is worth it to switch to it. Add more ram to the embedded board and welcome to the future.
- deleted 2y ago
  [deleted]
- simoncion 2y ago
  > The logging alone is worth it to switch to it. Weird. In my professional life, I've found systemd's logging to be woefully lacking when compared to systems that just run every daemon through syslog (or -even more retro- "each daemon writes stdout & stderr to disk, which then gets sent to your log sink of choice"). What I usually find is that a ton of information never, ever makes it into journalctl, so I always need to go to the actual log files on disk. [0] On top of that, journalctl just fucking giving up when logs have been rotated is totally bogus. (If this has been fixed, then the fix hasn't made its way to the systems I work with.) ALSO, have they fixed the issue where minor data corruption just totally fucks your log file to death... rather than what happens with regular log files and you get an unreadable section that's bracketed by good data? [0] Yes, you could reasonably say "Well, those services are not correctly configured to log to journalctl!". But my point here is that I see this happen so often that journalctl is like my tenth stop for log messages, rather than my first.
phendrenad2 2y ago
Wait so systemd is a problem for embedded Linux because... it uses 250MB of RAM? So it's only really a problem for systems where using a mainstream distro probably wouldn't even be a consideration anyway (yocto seems to be popular, as mentioned above). I'm not seeing a strong argument here. At the risk of inviting a parade of "actually, one time at band camp I saved the day with a first-gen raspberry pi zero", how many embedded systems in 2024 are really RAM-limited?
- detaro 2y ago
  > how many embedded systems in 2024 are really RAM-limited? A lot. Plenty embedded Linux stuff still ships with 256 MB or 512 MB RAM, and the wishlist for features in the software running on top is always getting longer than initially planned... But of course the average has moved up, and it's not unusual to see systemd in embedded systems either (while I don't have a number at hand, the 250MB number seems off to me). The space is big, and different constraints and rules at different points in there.
  - cyanydeez 2y ago
    Seems like a complaint the author could help solve by extending the general purpose systems to their choice of hardware.
  - metalforever 2y ago
    I remember when I used to run gnome 2 (i.e. mate) on a machine with 256mb of ram. It was a full experience and it worked with youtube videos and so on. What are we even doing .
    - BeefWellington 2y ago
      Sure, back when videos were at most 480p/720p that's feasible. I'm not saying software is any more efficiently written these days but I do think it's important to recognize that just the act of pushing more pixels on its own requires more RAM.
oceanplexian 2y ago
I used it heavily at work. Then moved to a shop that does exclusively containers and realized how incredibly immature engineers’ understanding of the problem space is. Container startup ordering, startup dependency management, socket handling and all things timing and network related are frankly a complete mess in modern infrastructure. Even, surprisingly in well maintained projects like k8s. I find myself shaking my head at problems solved by systemd a decade ago far too often.
cyberax 2y ago
> With care, though, a minimal installation of systemd does run on a low-resource ARM board like the Pi 3. In fact, it will run on a board with 1Gb RAM, perhaps even lower. Come fucking on. I ran systemd on a 32Mb MIPS board (RouterBoard from Mikrotik) just fine. The RES column is incredibly misleading in this case, because systemd is the first app in the system. So it gets charged for loading the entire glibc and the supporting cast of libraries. If you want to look at the true usage, check the RssAnon value: > root@kmipsrv:/proc/1# cat /proc/1/status | grep RssAnon > RssAnon: 3220 kB You can also pare this down a bit. Similarly for journald: > root@kmipsrv:/proc/1# cat /proc/231/status | grep RssAnon > RssAnon: 640 kB So in reality, systemd works just fine for the vast majority of embedded platforms. If you can spare 3Mb of RAM, then you can run a full-blown dependency management system with reliable recovery, logging integration, and event-driven device management. Soon with support for verified boot with hardware root-of-trust.
- amelius 2y ago
  This should be the top comment.
- synergy20 2y ago
  systemd itself is also large in storage size(8MB?), which is why openwrt has its own procd replacements, and yocto is not default to systemd either due to this. additionally, systemd does not work well with musl which is very popular in embedded linux.
  - cyberax 2y ago
    systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc. The minimum binaries size that can be obtained by trivial removals is around 2Mb. OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb. > systemd does not work well with musl which is very popular in embedded linux. Systemd can work with musl with a small compatibility shim, I did it about 9 years ago. Somebody did that again: https://catfox.life/2024/09/05/porting-systemd-to-musl-libc-powered-linux/ https://catfox.life/2024/09/05/porting-systemd-to-musl-libc-...
johnea 2y ago
The article fails to mention another aspect of systemd that interferes with emmbedded dvelopment: it's utility applications (primarily systemctl, but all the others as well) are intended to be run on the booted system for which the control is being performed. When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media. I've never liked it, I still don't like it, and I think the number of people in this camp is understated by the article. However, I am still running it. As other posts have mentioned, switching distributions is a major hasstle, especially if you've built tools using the distro's architecture. For me, this is archlinux. Although, I am in the process of testing and migrating to void linux. Which is systemd free, and hosts ARM and x86 binary package repositories.
- stefanha 2y ago
  Can you be specific about what cannot be configured on mounted embedded media? systemd follows the drop-in config file model where configuration snippets are placed into directories (like .d/ directories in Debian). It should not be necessary to run utilities from the target system in order to configure it on mounted media. Drop unit files for services, sockets, filesystem mounts, timers, etc onto the mounted media and they will be detected when the target system boots.
- blucaz 2y ago
  > When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media. systemctl --root /path/to/sd/card/ <...>
- poettering 2y ago
  Most of our tools that adjust files work really well offline. systemd-tmpfiles, systemd-sysusers, systemctl and so on all support --root= and --image=. I really don't know what the problem is suppoaed to be. Lennart
cbmuser 2y ago
The memory comparison isn’t really fair as SysVInit relies on external utilities for starting and stopping processes. At the very least, SysVInit requires a shell for running init scripts which systemd doesn’t need for native service files.
- deleted 2y ago
  [deleted]
- Spooky23 2y ago
  Real embedded developers wouldn’t stand for the overhead of modern stuff like SysVInit.
  - bmicraft 2y ago
    Those 'real developers' you're talking about wouldn't stand for the overhead of a kernel either
hansvm 2y ago
Not really embedded, but one of the easier classes of wins I've had over the years is moving prod servers to a non-systemd Gentoo. Random (rare) 200ms latency spikes and whatnot just disappear, and the system jitter drops down to the tens of microseconds range, significantly improving the qualitative performance of basically every piece of software I care about.
- BenjiWiebe 2y ago
  Did you ever dig in to see why that changed? I'm surprised that in a steady-state (not booting or restarting services) systemd/not-systemd would have an impact.
  - hansvm 2y ago
    A little bit. The tldr; is that there are a lot of periodic activities including log rotation, automatic updates, time syncing, waking up to check if such things were currently needed, .... Those are generally nice (aside from automatic updates, which are sometimes good but I think usually more harmful than whatever they're fixing), but systemd doesn't know anything about what my userspace demands on the hardware are, and it's generally happy to hog system resources if it doesn't do so for long. For anything quasi-realtime, I can certainly hunt down and squash everything systemd is running at the wrong time, but in practice it's much easier to install something else, ensure no processes are running other than the ones I care about, and move those concerns to userspace.
    - howtofly 2y ago
      > For anything quasi-realtime, I can certainly hunt down and squash everything systemd is running at the wrong time, but in practice it's much easier to install something else, ensure no processes are running other than the ones I care about, and move those concerns to userspace. Should these issues be fixed by CPU/IO schedulers? Are there any systemd realtime tasks involved?
      - hansvm 2y ago
foul 2y ago
Meh, we are not in 2021 or 2018 anymore. Eudev and elogind ARE mantained and used in voidlinux and gentoo, and anyways being that embedded it doesn't seem clear to me where would the "hard dependency of a very essential software on some systemd-thingd" happen.
- nubinetwork 2y ago
  https://www.gentoo.org/support/news-items/2021-08-24-eudev-retirement.html https://www.gentoo.org/support/news-items/2021-08-24-eudev-r...
  - foul 2y ago
    https://github.com/eudev-project/eudev https://github.com/eudev-project/eudev https://github.com/elogind/elogind https://github.com/elogind/elogind
jcalvinowens 2y ago
I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM. If I'm on a tiny machine where I care about 8MB RSS, I'm not running Linux, I'm running Zypher or FreeRTOS.
- avaldez_ 2y ago
  Small typo: Zephyr https://zephyrproject.org/ https://zephyrproject.org/
- blueflow 2y ago
  That "occasional weird problem" is because systemd is not designed to be used with other software. While you technically still have the choice to roll your own non-systemd initramfs, it will be an uphill battle.
  - noncoml 2y ago
    Not uphill at all; try buildroot
  - jcalvinowens 2y ago
    > That "occasional weird problem" is because systemd is not designed to be used with other software. That's not true, systemd has an explicitly documented and supported initrd interface: https://systemd.io/INITRD_INTERFACE/ https://systemd.io/INITRD_INTERFACE/ It's actually really easy, there's rarely a reason for the initrd to be more complex than a single 50-line busybox script on an embedded device with built-in drivers.
    - blueflow 2y ago
      "really easy" except for the hotplug events that don't get to systemd and cause your problem.
blueflow 2y ago
There is another aspect to the memory footprint: Does systemd fit into the L1 cache? Does sysvinit? Both x86 and Linux (page cache) rely on caches for performance.
- worthless-trash 2y ago
  I'm not sure that fitting a init system in L1 cache should be a priority when it would be swapped out almost immediately on doing further work, aka the purpose of the device.
rettichschnidi 2y ago
We (https://github.com/husqvarnagroup/smart-garden-gateway-public https://github.com/husqvarnagroup/smart-garden-gateway-publi...) are using systemd on devices with 128 MB of RAM and are happy with it. It solves so many problems one usually has to fiddle with, totally worth a few MBs of RAM.
cozzyd 2y ago
I run systemd-based Debian just fine on 512 MB RAM BeagleBoneBlack SBCs with 4GB emmcs. Now emacs, that's a problem due to the large install size.
- TacticalCoder 2y ago
  > Now emacs, that's a problem due to the large install size. I was running Emacs on a 486 with something like 16 MB of RAM. Don't remember the HDD size. The Emacs executable I built from source is about 28 MB. Honestly I don't know what else is needed besides that executable that'd be really big. Is it really that problematic that it wouldn't work with 512 MB of RAM and 4 GB of storage?
  - cozzyd 2y ago
    I think the problem is the Debian packaging, which has hundreds of MB of elisp or something. https://packages.debian.org/sid/emacs-common https://packages.debian.org/sid/emacs-common (yes, it does fit, but it takes up way more space than I'd like...)
hactually 2y ago
> I wish I knew the solution to this problem Ooh ooh, I know! It's to improve systemd for the use-cases outlined and build a really great open source init/process manager.
- simoncion 2y ago
  The systemd maintainers are pretty particular about things that aren't a "supported use case", so even if you bother to do the work, you stand a solid chance of having your work be wasted. In my professional experience, "unsupported use case" effectively means "I don't want to work on that". [0] Also, at $DAYJOB, we run into mysterious systemd failures and misbehaviors at least once a year (usually for things that should be super straightforward). Every one of these failures and misbehaviors we've run into has been met with a "Wow. That's weird. Sucks to be you, I guess." with a dash of "No, we won't be updating the documentation to mention that." if the actual behavior is contrary to the documentation. In short, while it may be true (I really don't know) that the systemd project welcomes changes and patches, IME it's pretty clearly true that if you're working on a problem that the project management doesn't understand, and/or doesn't want to think through, your work won't be considered. [0] [0] Which, fine, it's not my project, so I don't get to tell you what to work on. Doesn't mean I have to like it or recommend that people use it.
  - throwaway2037 2y ago
    > we run into mysterious systemd failures and misbehaviors at least once a year Can you share an example?
    - simoncion 2y ago
      > Can you share an example? The most detail my NDA permits me to get into is that our most recent failure was that the systemd cron replacement gets in a state where it refuses to ever schedule a scheduled task and that the systemd folks were like "aw, shucks" about it.
    - Yizahi 2y ago
      An example from my company - sometimes on a few of the many thousands of identical devices in field something fails due to bug or hw failure. After recovering the device we get the dump of all info we deem interesting, including all logs and journals. In a small but not insignificant numbers of such archives the journal has multihour gaps in the logs. And they are gaps, so there are logs before the gaps and after. I think we never succeeded to reproduce or fix this issue in the lab, and relied on duplicating logs to the file on the device fylesystem (not optimal due to storage size and wear of the ssd memory). PS: personally I like systemd more that the script mess which preceded it. But the are some outstanding issues with it to be improved.
neuroelectron 2y ago
As for a solution to this, wouldn't it be possible to use systemd to build a targeted init script? Use the expertise embedded in systemd to bring up the system you're targeting, recording how it does that and then removing systemd, leaving the artifacts? Why keep this huge monolithic system running in the background after it has served its purpose? Also, with ai, would it be possible to redraw the lines within systemd between the separate domains/concerns, then redistribute the functions back to where they conceptually belong? Then you can refine those changes to make them more standardized and universal instead of interdependent and centralized.
greatgib 2y ago
I'm quite annoyed by the bland statement in the introduction of this article: people make about it don’t stand up to much scrutiny, even the technical ones Basically the author says that everyone else has stupid rants that doesn't stand but himself has valid claims so that in the end we have a proof that there are a few valid technical rants! That being said, the main problem with systemd from the beginning is that it is a cancer. It was explicitly designed to prevent as much as possible the modularity and intercompatibility. Kind of opposite to the POSIX way even I think. Despite what is pretended, doing the things would probably not have impacted features, like boot speed but it was key for them to shove through our throat the usage of systemd in a "all or nothing" way. And udev or logind are good example of that.
- reanimus 2y ago
  I think it's fair to say the majority of people don't really care about any of those things as much as having a system that just works -- which was part of their whole argument regarding why package maintainers love it so much and end users really don't tend to care.
  - kytazo 2y ago
    I would assume people using Linux more or less care to an extend
- Vilian 2y ago
  And systemd replace all POSIX init system because they were all bad or systemd do the same thing better or faster, your point is a good argument against following POSIX religiously
msarnoff 2y ago
I’ve never used it as an init replacement, but runit (https://smarden.org/runit/runit.8 https://smarden.org/runit/runit.8) is great for supervising applications and services. - It’s built into BusyBox and is very lightweight - Configuration is dead simple (everything is basically shell scripts in the filesystem; no special config language) - Includes logging functionality with rotation - Easily controllable from other applications using named pipes - It’s an almost perfect embodiment of the Unix philosophy: a series of small, single-purpose executables (svlogd, runsv, runsvdir, etc), with no strange config file formats or protocols. Regarding udev, it’s entirely possible to run an embedded system without it if your device has a fixed set of peripherals, doesn’t need to handle hotplugged hardware, and uses a fixed set of kernel modules. In Yocto, you can define a dummy device manager recipe and boom, no udev. As a bonus, removing udev can shave several seconds off your boot time.
- binkHN 2y ago
  For what it's worth, runit is the default on Void Linux.
klysm 2y ago
I've had very nice experiences using systemd for developing embedded systems. Laying out all the unit files and just slapping the files in an overlay makes it super simple. The RAM usage isn't being calculated correctly. You can easily run systemd on very limited systems.
teunispeters 2y ago
If it can run X11 on a directly connected display, it's probably not embedded spaces ;). Don't take this comment too seriously, however raspberry pi is definitely at the "more desktop-like" end of the embedding pool. As for systemd, it's an interesting set of tools, and one can if one wishes pick which ones are active in a system, if that system is complex enough to be running Linux (or other compatible OS) to begin with.
m463 2y ago
I think what he's running into is "the unix philosophy". It is basically about small tools, limited in scope, that solve a problem surgically and comprehensively. They work together by mixing and matching to solve the problem. systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing. I don't know, it is sort of like busybox, replacing other system components in a limited Minimum Viable Product way, but more like limited function, not limited resources.
- cyberax 2y ago
  > I think what he's running into is "the unix philosophy". "Unix philosophy" is throwing crap at the problem, until it kinda-sorta works on the developer's machine.
- oguz-ismail 2y ago
  > systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing And it's slow. Windows 11 boots in 5 seconds on my laptop, Ubuntu 22 used to take about a minute before I finally made the switch.
  - dralley 2y ago
    That almost certainly has little to do with systemd itself, and more to do with which services are enabled on boot.
    - lmm 2y ago
      It's due to systemd in that systemd makes it too fiddly to figure out or change which services are enabled on boot. (I used to know a way to disable certain services on boot under systemd, but it doesn't work any more, and I've reached a state of learned helplessness at this point)
      - vetinari 2y ago
        `systemctl disable` and `systemctl mask` work today just like they worked decade and half ago.
iluvcommunism 2y ago
I like systemd sometimes. Other times it confuses me. Particularly the way cgroups work. The way splicing works. Why oracle mem usage is still uncontrollable when I set user splice restrictions. Idk, I’m still learning it. I don’t like the terminology it uses sometimes too like “snapshot.” When I first read about systemd snapshot I was like oh cool you can snapshot the system. No. Also some esoteric things it supposedly supports don’t work sometimes. I find the idea of systemd targets and fstab having anything to do with each other anathema. Idk.
0xbadcafebee 2y ago
Systemd, Wayland, DBus, and other modern Linux "solutions" are all problematic for the same reason: they are complex monoliths that are extremely difficult to replace, in part or in whole. Systems often won't work without them, because the entire ecosystem has grown dependent on them. You literally can't run a modern desktop without a bunch of "shims" whose sole purpose is to fake being systemd. The best designs, sadly, don't win out. The ones that win out are the ones that serve the interests of the largest players. And the largest players are usually RedHat, Google, and other vendors, who just want to get "their" needs met and move on. They couldn't give a fig about compatibility with some other random tiny project, or an embedded device. And those people working on tiny projects aren't in the room when "standards" are decided for the entire community. There are some hold-outs. Slackware went a long time without it, but next release will unfortunately use systemd. I've run Alpine Linux (systemd-free, so far) as my laptop OS for years. I'll probably switch to another distro that's more user friendly, is easier to contribute to. But it's embarrassing and annoying that we have to pretend to be systemd-compatible just to run a desktop. We should call it "GNU/Systemd/Linux" since it's now one big monolith.
- thnkman 2y ago
  Where have you heard that Slackware will be using systemd next release? I have been using slackware for years, and this is news to me.
  - pwg 2y ago
    There's nothing named "systemd" in the FILE_LIST [1] for Slack64-current (which is what eventually becomes "the next release" at the appropriate time. So it would seem the GP has been taken in by some rumor somewhere. [1] https://ftp.ussg.indiana.edu/linux/slackware/slackware64-current/slackware64/FILE_LIST https://ftp.ussg.indiana.edu/linux/slackware/slackware64-cur...
    - 0xbadcafebee 2y ago
      I thought I saw in the last release a note that the next one would come with systemd. It could be I hallucinated it. Maybe I'm becoming a real LLM!
worthless-trash 2y ago
Excellent plan, go to the other inits, bug bounties are what I do over christmas and init systems and hand rolled init scripts are a great source of security flaws. Taking notes on which distros dont run systemd, for when I want to make some money over christmas.
- DaSHacka 2y ago
  No need to stop at non-systemd inits! https://security.gentoo.org/glsa https://security.gentoo.org/glsa https://security.archlinux.org/issues/all https://security.archlinux.org/issues/all ^ (ctrl+f systemd) https://ubuntu.com/security/cves?package=systemd&limit=100 https://ubuntu.com/security/cves?package=systemd&limit=100 https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Systemd https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Systemd
typ 2y ago
I have no philosophy issue with systemd. The biggest show-stopper (on embedded Linux) for me is its install size. We have edge devices with only 64M RAM and 128M NAND flash. For an OTA update (via Bluetooth sometimes) only ~20M in total, adding ~5M systemd plus its dependencies doesn't make sense. 5MiB is roughly equivalent to the entire Linux kernel image on those devices. Among the components of systemd, the part that has the lowest added value for our use is dbus. It's designed primarily for desktops and useless for most of our systems, but unfortunately, it is also a hard requirement for systemd.
dzannetandreevn 2y ago
[dead]
nrdvana 2y ago
Try s6 or runit, and mdev from busybox. Mdev is not a "full featured" udev replacement, but that's sort of the point. It does just enough to get the job done and is extensible with scripts. If you want to do the size of embedded where this matters, you also probably want a smaller libc like musl, and then buildroot and busybox are your friends.
richardfey 2y ago
I think author is missing on the Devuan-maintained udev fork.
Jean-Papoulos 2y ago
"I'm using a thing in a way it's not meant to be used and it sucks" Is this even a serious post.
- robertlagrant 2y ago
  It is a serious post; you've just summarised it appallingly.
m0llusk 2y ago
It is unfortunate that after all this time the basic reasons why systemd was introduced and has been so successful are pervasively misunderstood. > The majority of Linux users are uninterested in the pros and cons of systemd. A small number are violently opposed to it, and a small number are violently opposed to those who are opposed to it. Starting out by framing systemd in terms of opposition deliberately steers attention away from the issues that drive its adoption. The tone is moderate, but this is not good faith argumentation. > I think there’s little argument that the main target for systemd is a general-purpose computer, with a modern, integrated graphical desktop That is one of the targets. The main target is servers that run the web of services typically required to back up modern applications. What if an application depends on a resource server which itself depends on a remote filesystem being mounted and an event service running. Theoretically that can work with System V init, but how tricky is that to put in place and make robust? Administrators for systems vending complex interdependent services are the main target and always have been. > Unfortunately, what makes systemd good for general-purpose desktop applications Get it wrong and then run with it. Failing to understand what is driving adoption of systemd leads to arguments about it that don't really go anywhere. > The more fundamental problem is that the people who most like systemd are distribution managers. Sure, there are administrators who like it, and defend it vigorously; but most end users and administrators don’t really care. Not distribution managers, but system administrators. They are the ones with the complex needs that systemd serves. As such they advocate for it and because of the lack of working alternatives distribution managers switch to it in order to please the largest number of clients. Thinking of systemd in terms of desktop systems and distribution managers means misunderstanding why it is there in the first place.
kj4ips 2y ago
TBH, my biggest gripe is that systemd doesn't play nice with musl or ulibc, I would like to not have to ship a cron, a supervisor, a xinetd equivelant, and a binary just to react to inotify/dnotify events. Though openrc how has a supervisor.
M95D 2y ago
Big systemd on embedded devices is actually A GOOD THING! Imagine what Gentoo, Buildroot/OpenWrt or even Busybox alone could do with so much memory after removing all that bloated OS!

208 points by synergy20 2y ago

is_taken 2y ago

You could checkout void and alpine.

userbinator 2y ago

The BSDs may be more to your liking. They're a lot more minimalist than Linux now.

Aurornis 2y ago
In the embedded world, BSP/driver support is the main thing that matters. Linux is undefeated in that area for embedded purposes.
- moron4hire 2y ago
  I thought there was a compatibility layer available that made it possible to run Linux drivers on at least FreeBSD.
  - CorrectHorseBat 2y ago
    That's for userland, not drivers. Linux doesn't have a stable interface for drivers so a compatibility layer would be a nightmare to maintain.
    - moron4hire 2y ago
      That's Linuxulator, but there's also LinuxKPI: https://wiki.freebsd.org/LinuxKPI https://wiki.freebsd.org/LinuxKPI
- bluGill 2y ago
  In embedded you should choose hardware that works. Should is key, all too often the, hardway guys don't ask they just give software something.
  - nomel 2y ago
    > In embedded you should choose hardware that works. I've never seen it approached with that kind of incompetence, in my professional life. Embedded is a system thing, not a hardware thing. You pick the system that will work. Ignoring the software side of things is ignoring the majority of the problem in embedded work. I would have agreed with you 20 years ago, but it's 2024, and why RPI is so often used for prototyping.
    - bluGill 2y ago
      Sorry, wrote the original comment on my phone. The sentiment is close enough, but generally things are not quite that bad. I've seen hardware ask, but the person they asked didn't know how to check how much memory we actually needed. Hardware also failed to tell that person that half the memory they were asking about was dedicated to the GPU and thus not available for application code. I've also seen them choose hardware that seems to have great drivers and they can point to others in our industry who are shipping with that, only to latter hit a major bug that means we can't use it without that fixed - but we are too small a customer to be able to demand they fix bugs that only affect us. RPI is often useful for prototyping because it is cheap enough, but if you are not a very large company nobody will talk to you. This is both you can't get support from Broadcom, and worse your supply management cannot get contract signed (for small companies this doesn't matter you take what you can get, but larger companies often demand better support than they can get from pi)

solarkraft 2y ago

> The more fundamental problem is that the people who most like systemd are distribution managers. (...) a distribution manager doesn’t have to maintain, integrate, and support a whole bunch of system utilities from different sources – systemd provides everything in one huge build. That’s actually a great lesson in optimizing your product for adoption.

exe34 2y ago
it's also why corporate software is so shit - it's shiny in all the ways it needs to be shiny for management to make the decision, but it's a pain for the actual users. the trick in both cases is to know your customers!
- oblio 2y ago
  Turns out, many people use systemd after what, 15? years of doomsday predictions :-)
  - stepupmakeup 2y ago
    Unfortunately not by choice
  - fsckboy 2y ago
    i don't notice anything good systemd does for me, just where it interferes, and that's a constant annoyance, a few days a week.
    - viraptor 2y ago
      Can you link some upstream issues you raised about the big annoyances?
      - exe34 2y ago
        not many people enjoy bikeshedding.
      - 2y ago

pnathan 2y ago

I can only say that the obvious answer here is to throw effort into supporting & improving Gentoo.

viraptor 2y ago

I'm conflicted on the name/issue. I don't have any issues with Linux on embedded systems, because I'd use a minimal init with that one single app it should be running. (Or as mentioned in other comment, build your minimal distro yocto-style) RPi is a low spec general computer at this point. The idea of running gnome on an embedded system just doesn't make sense in my mind... RPi took the form factor that was used for industrial control boards and made it into a usable desktop... which people use as a desktop (or home server). But it created a segment which really is neither. And a lot of the software just isn't written with that segment in mind. Maybe as it gets more popular, people will pay more attention. But currently it seems more of an expectations issue.

fyrn_ 2y ago

https://chimera-linux.org/ https://chimera-linux.org/ https://chimera-linux.org/docs/faq#what-is-the-projects-take-on-systemd https://chimera-linux.org/docs/faq#what-is-the-projects-take... Is going a really interesting direction with it's service management and login / seat management. Fwiw, I think it's author also has a more gounded perspective on systemd then many others. For Chimera the main issue is systemd's high volume use of gnulibc and gcc extensions, which is a porabiluty issue for Chimera which uses a LLVM toolchain and musl libc (and a bunch of really cool hardening and compiler engineering)

binkHN 2y ago
I like the direction Chimera is taking; the lead is an ex-Void maintainer and there's clearly a lot of inspiration from there.

colonwqbang 2y ago

This is anti-systemd FUD. Implying that applications which work without systemd are going to stop doing so. Suggesting that Linux distributions adopted systemd because it's good for distributors and not because users actually want it. We are "forced" to use systemd's udev implementation. Please.

felixgallo 2y ago
All of those things are trivially true. What are you disagreeing with?
- viraptor 2y ago
  There are extremely few use cases where your app may want to integrate with systemd. Out of those, you get again a small percentage that can't be trivially made optional. (99.9% is just optional notification and socket passing) Outside of large projects which are interested in full system integration like gnome, nobody breaks apps to not work without systemd.
- consteval 2y ago
  Except none of these are true, they're just things people have made up to conceal the fact they oppose systemd for ideological reasons. There's little to no technical arguments against systemd - it's fast, much more robust than what came before it, and services 99% of usecases. Users DID want systemd. Ask any sys admin, most much prefer systemd. Also systemd doesn't "force" anything. Most distributions don't even ship all systemd projects, just a couple. systemd-init IS small. The idea that systemd is a big ole monolith is just not true, and one glance at the git repo reveals that.
  - felixgallo 2y ago
    oh definitely not, and it's a matter of record. Red Hat and Poettering wanted systemd in order to control more of the platform; this is why they absorbed in udev (which had been a separate repo), came out saying that there are certain parts of the system that, despite all this being open source, were forbidden to modify or replicate, and stacked the debian voting. Systemd at the time sucked, incredibly, even worse than it does today, and routinely bricked or crashed systems. The adoption was forced and 100% political. systemd is, of course,today a continuing shambling feature factory behemoth in which hundreds of product managers try to shoehorn in more mandatory features in order to cement grip on the platform. That's why you get this ridiculous dns nonsense, this ridiculous container nonsense, this ridiculous cron nonsense, hostnamed (!), the list is endless. The technical argument is that it's a giant tasteless bag of ever increasing poorly implemented scope. The rest of Unix, by comparison, is generally not that way. If you pick up a BSD or even an alpine, you'll find that you don't need a bunch of badly written, hacked together garbage in order to get your work done. The system can be entirely readable and repeatable without cramming a bunch of cruft and poor decisions into pid 1. The idea that systemd doesn't 'force' anything is, of course, hilarious. Systemd is designed to try to be the ultimate mandatory dependency for essentially everything on the system. That's the way Red Hat wanted it to be, and that's the way it is. Every time someone mentions systemd, some random apologist dutifully trots out the idea that systemd-init is small and therefore systemd is not a monolith, checkmate. Of course, journald, libudev, localed, logind, hostnamed, homed, networkd, resolevd, systemd-boot, systemd-bsod, systemd-nspawn, timedated, timesyncd, tmpfiles, udevd, and all of the other array of dumbass bolted-on second-system-effect-driven product-managed nonsense somehow don't get mentioned.

transpute 2y ago

OpenEmbedded/Yocto, Devuan and Gentoo provide multiple init systems. systemd CVEs: https://ubuntu.com/security/cves?package=systemd&limit=100 https://ubuntu.com/security/cves?package=systemd&limit=100 Rust PoC: https://github.com/KillingSpark/rustysd https://github.com/KillingSpark/rustysd > Rustysd is a service manager that tries to replicate systemd behaviour for a subset of the configuration possibilities. It focuses on the core functionality of a service manager, not requiring to be PID1 (aka init process).. core systemd functionality is not very hard to replicate.. the advantages of having a systemd-like service manager can be brought to many other platforms.. (like alpine linux, commonly used in docker containers and small vms) and freebsd.

dcsommer 2y ago
How many systemd CVEs are memory or thread safety related?
- fn-mote 2y ago
  I can't tell if this is rhetorical but... it looks like quite a few on the list. (Note: the CVE list is long but it looks inflated / full of not-serious issues.) * CVE-2022-3821 : off by one error leading to buffer overflow * CVE-2022-2526 : use after free * CVE-2021-33910 : "Memory Allocation with an Excessive Size Value" (not sure if this qualifies... not interested enough to read the source)
- silverliver 2y ago
  What do memory and thread safety have to do with the project's stated goals? I for one would love to avoid systemd's arbitrary requirements while retaining compatibility with upstream (e.g. PID must be 1).
- 1oooqooq 2y ago
  I'd bet you get more CVEs and actual remote exploit entry points by systemd forcing mdns, bonjour, upnp and other zero conf hacks just because that was the work the systemd team was doing in rh before.

Mave83 2y ago

For me as a user, systemd is far better than old init systems. The logging alone is worth it to switch to it. Add more ram to the embedded board and welcome to the future.

deleted 2y ago
[deleted]
simoncion 2y ago
> The logging alone is worth it to switch to it. Weird. In my professional life, I've found systemd's logging to be woefully lacking when compared to systems that just run every daemon through syslog (or -even more retro- "each daemon writes stdout & stderr to disk, which then gets sent to your log sink of choice"). What I usually find is that a ton of information never, ever makes it into journalctl, so I always need to go to the actual log files on disk. [0] On top of that, journalctl just fucking giving up when logs have been rotated is totally bogus. (If this has been fixed, then the fix hasn't made its way to the systems I work with.) ALSO, have they fixed the issue where minor data corruption just totally fucks your log file to death... rather than what happens with regular log files and you get an unreadable section that's bracketed by good data? [0] Yes, you could reasonably say "Well, those services are not correctly configured to log to journalctl!". But my point here is that I see this happen so often that journalctl is like my tenth stop for log messages, rather than my first.

phendrenad2 2y ago

Wait so systemd is a problem for embedded Linux because... it uses 250MB of RAM? So it's only really a problem for systems where using a mainstream distro probably wouldn't even be a consideration anyway (yocto seems to be popular, as mentioned above). I'm not seeing a strong argument here. At the risk of inviting a parade of "actually, one time at band camp I saved the day with a first-gen raspberry pi zero", how many embedded systems in 2024 are really RAM-limited?

detaro 2y ago
> how many embedded systems in 2024 are really RAM-limited? A lot. Plenty embedded Linux stuff still ships with 256 MB or 512 MB RAM, and the wishlist for features in the software running on top is always getting longer than initially planned... But of course the average has moved up, and it's not unusual to see systemd in embedded systems either (while I don't have a number at hand, the 250MB number seems off to me). The space is big, and different constraints and rules at different points in there.
- cyanydeez 2y ago
  Seems like a complaint the author could help solve by extending the general purpose systems to their choice of hardware.
- metalforever 2y ago
  I remember when I used to run gnome 2 (i.e. mate) on a machine with 256mb of ram. It was a full experience and it worked with youtube videos and so on. What are we even doing .
  - BeefWellington 2y ago
    Sure, back when videos were at most 480p/720p that's feasible. I'm not saying software is any more efficiently written these days but I do think it's important to recognize that just the act of pushing more pixels on its own requires more RAM.

oceanplexian 2y ago

I used it heavily at work. Then moved to a shop that does exclusively containers and realized how incredibly immature engineers’ understanding of the problem space is. Container startup ordering, startup dependency management, socket handling and all things timing and network related are frankly a complete mess in modern infrastructure. Even, surprisingly in well maintained projects like k8s. I find myself shaking my head at problems solved by systemd a decade ago far too often.

cyberax 2y ago

> With care, though, a minimal installation of systemd does run on a low-resource ARM board like the Pi 3. In fact, it will run on a board with 1Gb RAM, perhaps even lower. Come fucking on. I ran systemd on a 32Mb MIPS board (RouterBoard from Mikrotik) just fine. The RES column is incredibly misleading in this case, because systemd is the first app in the system. So it gets charged for loading the entire glibc and the supporting cast of libraries. If you want to look at the true usage, check the RssAnon value: > root@kmipsrv:/proc/1# cat /proc/1/status | grep RssAnon > RssAnon: 3220 kB You can also pare this down a bit. Similarly for journald: > root@kmipsrv:/proc/1# cat /proc/231/status | grep RssAnon > RssAnon: 640 kB So in reality, systemd works just fine for the vast majority of embedded platforms. If you can spare 3Mb of RAM, then you can run a full-blown dependency management system with reliable recovery, logging integration, and event-driven device management. Soon with support for verified boot with hardware root-of-trust.

amelius 2y ago
This should be the top comment.
synergy20 2y ago
systemd itself is also large in storage size(8MB?), which is why openwrt has its own procd replacements, and yocto is not default to systemd either due to this. additionally, systemd does not work well with musl which is very popular in embedded linux.
- cyberax 2y ago
  systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc. The minimum binaries size that can be obtained by trivial removals is around 2Mb. OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb. > systemd does not work well with musl which is very popular in embedded linux. Systemd can work with musl with a small compatibility shim, I did it about 9 years ago. Somebody did that again: https://catfox.life/2024/09/05/porting-systemd-to-musl-libc-powered-linux/ https://catfox.life/2024/09/05/porting-systemd-to-musl-libc-...

johnea 2y ago

The article fails to mention another aspect of systemd that interferes with emmbedded dvelopment: it's utility applications (primarily systemctl, but all the others as well) are intended to be run on the booted system for which the control is being performed. When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media. I've never liked it, I still don't like it, and I think the number of people in this camp is understated by the article. However, I am still running it. As other posts have mentioned, switching distributions is a major hasstle, especially if you've built tools using the distro's architecture. For me, this is archlinux. Although, I am in the process of testing and migrating to void linux. Which is systemd free, and hosts ARM and x86 binary package repositories.

stefanha 2y ago
Can you be specific about what cannot be configured on mounted embedded media? systemd follows the drop-in config file model where configuration snippets are placed into directories (like .d/ directories in Debian). It should not be necessary to run utilities from the target system in order to configure it on mounted media. Drop unit files for services, sockets, filesystem mounts, timers, etc onto the mounted media and they will be detected when the target system boots.
blucaz 2y ago
> When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media. systemctl --root /path/to/sd/card/ <...>
poettering 2y ago
Most of our tools that adjust files work really well offline. systemd-tmpfiles, systemd-sysusers, systemctl and so on all support --root= and --image=. I really don't know what the problem is suppoaed to be. Lennart

cbmuser 2y ago

The memory comparison isn’t really fair as SysVInit relies on external utilities for starting and stopping processes. At the very least, SysVInit requires a shell for running init scripts which systemd doesn’t need for native service files.

deleted 2y ago
[deleted]
Spooky23 2y ago
Real embedded developers wouldn’t stand for the overhead of modern stuff like SysVInit.
- bmicraft 2y ago
  Those 'real developers' you're talking about wouldn't stand for the overhead of a kernel either

hansvm 2y ago

Not really embedded, but one of the easier classes of wins I've had over the years is moving prod servers to a non-systemd Gentoo. Random (rare) 200ms latency spikes and whatnot just disappear, and the system jitter drops down to the tens of microseconds range, significantly improving the qualitative performance of basically every piece of software I care about.

BenjiWiebe 2y ago
Did you ever dig in to see why that changed? I'm surprised that in a steady-state (not booting or restarting services) systemd/not-systemd would have an impact.
- hansvm 2y ago
  A little bit. The tldr; is that there are a lot of periodic activities including log rotation, automatic updates, time syncing, waking up to check if such things were currently needed, .... Those are generally nice (aside from automatic updates, which are sometimes good but I think usually more harmful than whatever they're fixing), but systemd doesn't know anything about what my userspace demands on the hardware are, and it's generally happy to hog system resources if it doesn't do so for long. For anything quasi-realtime, I can certainly hunt down and squash everything systemd is running at the wrong time, but in practice it's much easier to install something else, ensure no processes are running other than the ones I care about, and move those concerns to userspace.
  - howtofly 2y ago
    > For anything quasi-realtime, I can certainly hunt down and squash everything systemd is running at the wrong time, but in practice it's much easier to install something else, ensure no processes are running other than the ones I care about, and move those concerns to userspace. Should these issues be fixed by CPU/IO schedulers? Are there any systemd realtime tasks involved?
    - hansvm 2y ago

foul 2y ago

Meh, we are not in 2021 or 2018 anymore. Eudev and elogind ARE mantained and used in voidlinux and gentoo, and anyways being that embedded it doesn't seem clear to me where would the "hard dependency of a very essential software on some systemd-thingd" happen.

nubinetwork 2y ago
https://www.gentoo.org/support/news-items/2021-08-24-eudev-retirement.html https://www.gentoo.org/support/news-items/2021-08-24-eudev-r...
- foul 2y ago
  https://github.com/eudev-project/eudev https://github.com/eudev-project/eudev https://github.com/elogind/elogind https://github.com/elogind/elogind

jcalvinowens 2y ago

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience. I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683 https://github.com/systemd/systemd/issues/34683 Memory use is irrelevant to me: every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM. If I'm on a tiny machine where I care about 8MB RSS, I'm not running Linux, I'm running Zypher or FreeRTOS.

avaldez_ 2y ago
Small typo: Zephyr https://zephyrproject.org/ https://zephyrproject.org/
blueflow 2y ago
That "occasional weird problem" is because systemd is not designed to be used with other software. While you technically still have the choice to roll your own non-systemd initramfs, it will be an uphill battle.
- noncoml 2y ago
  Not uphill at all; try buildroot
- jcalvinowens 2y ago
  > That "occasional weird problem" is because systemd is not designed to be used with other software. That's not true, systemd has an explicitly documented and supported initrd interface: https://systemd.io/INITRD_INTERFACE/ https://systemd.io/INITRD_INTERFACE/ It's actually really easy, there's rarely a reason for the initrd to be more complex than a single 50-line busybox script on an embedded device with built-in drivers.
  - blueflow 2y ago
    "really easy" except for the hotplug events that don't get to systemd and cause your problem.

blueflow 2y ago

There is another aspect to the memory footprint: Does systemd fit into the L1 cache? Does sysvinit? Both x86 and Linux (page cache) rely on caches for performance.

worthless-trash 2y ago
I'm not sure that fitting a init system in L1 cache should be a priority when it would be swapped out almost immediately on doing further work, aka the purpose of the device.

rettichschnidi 2y ago

We (https://github.com/husqvarnagroup/smart-garden-gateway-public https://github.com/husqvarnagroup/smart-garden-gateway-publi...) are using systemd on devices with 128 MB of RAM and are happy with it. It solves so many problems one usually has to fiddle with, totally worth a few MBs of RAM.

cozzyd 2y ago

I run systemd-based Debian just fine on 512 MB RAM BeagleBoneBlack SBCs with 4GB emmcs. Now emacs, that's a problem due to the large install size.

TacticalCoder 2y ago
> Now emacs, that's a problem due to the large install size. I was running Emacs on a 486 with something like 16 MB of RAM. Don't remember the HDD size. The Emacs executable I built from source is about 28 MB. Honestly I don't know what else is needed besides that executable that'd be really big. Is it really that problematic that it wouldn't work with 512 MB of RAM and 4 GB of storage?
- cozzyd 2y ago
  I think the problem is the Debian packaging, which has hundreds of MB of elisp or something. https://packages.debian.org/sid/emacs-common https://packages.debian.org/sid/emacs-common (yes, it does fit, but it takes up way more space than I'd like...)

hactually 2y ago

> I wish I knew the solution to this problem Ooh ooh, I know! It's to improve systemd for the use-cases outlined and build a really great open source init/process manager.

simoncion 2y ago
The systemd maintainers are pretty particular about things that aren't a "supported use case", so even if you bother to do the work, you stand a solid chance of having your work be wasted. In my professional experience, "unsupported use case" effectively means "I don't want to work on that". [0] Also, at $DAYJOB, we run into mysterious systemd failures and misbehaviors at least once a year (usually for things that should be super straightforward). Every one of these failures and misbehaviors we've run into has been met with a "Wow. That's weird. Sucks to be you, I guess." with a dash of "No, we won't be updating the documentation to mention that." if the actual behavior is contrary to the documentation. In short, while it may be true (I really don't know) that the systemd project welcomes changes and patches, IME it's pretty clearly true that if you're working on a problem that the project management doesn't understand, and/or doesn't want to think through, your work won't be considered. [0] [0] Which, fine, it's not my project, so I don't get to tell you what to work on. Doesn't mean I have to like it or recommend that people use it.
- throwaway2037 2y ago
  > we run into mysterious systemd failures and misbehaviors at least once a year Can you share an example?
  - simoncion 2y ago
    > Can you share an example? The most detail my NDA permits me to get into is that our most recent failure was that the systemd cron replacement gets in a state where it refuses to ever schedule a scheduled task and that the systemd folks were like "aw, shucks" about it.
  - Yizahi 2y ago
    An example from my company - sometimes on a few of the many thousands of identical devices in field something fails due to bug or hw failure. After recovering the device we get the dump of all info we deem interesting, including all logs and journals. In a small but not insignificant numbers of such archives the journal has multihour gaps in the logs. And they are gaps, so there are logs before the gaps and after. I think we never succeeded to reproduce or fix this issue in the lab, and relied on duplicating logs to the file on the device fylesystem (not optimal due to storage size and wear of the ssd memory). PS: personally I like systemd more that the script mess which preceded it. But the are some outstanding issues with it to be improved.

neuroelectron 2y ago

As for a solution to this, wouldn't it be possible to use systemd to build a targeted init script? Use the expertise embedded in systemd to bring up the system you're targeting, recording how it does that and then removing systemd, leaving the artifacts? Why keep this huge monolithic system running in the background after it has served its purpose? Also, with ai, would it be possible to redraw the lines within systemd between the separate domains/concerns, then redistribute the functions back to where they conceptually belong? Then you can refine those changes to make them more standardized and universal instead of interdependent and centralized.

greatgib 2y ago

I'm quite annoyed by the bland statement in the introduction of this article: people make about it don’t stand up to much scrutiny, even the technical ones Basically the author says that everyone else has stupid rants that doesn't stand but himself has valid claims so that in the end we have a proof that there are a few valid technical rants! That being said, the main problem with systemd from the beginning is that it is a cancer. It was explicitly designed to prevent as much as possible the modularity and intercompatibility. Kind of opposite to the POSIX way even I think. Despite what is pretended, doing the things would probably not have impacted features, like boot speed but it was key for them to shove through our throat the usage of systemd in a "all or nothing" way. And udev or logind are good example of that.

reanimus 2y ago
I think it's fair to say the majority of people don't really care about any of those things as much as having a system that just works -- which was part of their whole argument regarding why package maintainers love it so much and end users really don't tend to care.
- kytazo 2y ago
  I would assume people using Linux more or less care to an extend
Vilian 2y ago
And systemd replace all POSIX init system because they were all bad or systemd do the same thing better or faster, your point is a good argument against following POSIX religiously

msarnoff 2y ago

I’ve never used it as an init replacement, but runit (https://smarden.org/runit/runit.8 https://smarden.org/runit/runit.8) is great for supervising applications and services. - It’s built into BusyBox and is very lightweight - Configuration is dead simple (everything is basically shell scripts in the filesystem; no special config language) - Includes logging functionality with rotation - Easily controllable from other applications using named pipes - It’s an almost perfect embodiment of the Unix philosophy: a series of small, single-purpose executables (svlogd, runsv, runsvdir, etc), with no strange config file formats or protocols. Regarding udev, it’s entirely possible to run an embedded system without it if your device has a fixed set of peripherals, doesn’t need to handle hotplugged hardware, and uses a fixed set of kernel modules. In Yocto, you can define a dummy device manager recipe and boom, no udev. As a bonus, removing udev can shave several seconds off your boot time.

binkHN 2y ago
For what it's worth, runit is the default on Void Linux.

klysm 2y ago

I've had very nice experiences using systemd for developing embedded systems. Laying out all the unit files and just slapping the files in an overlay makes it super simple. The RAM usage isn't being calculated correctly. You can easily run systemd on very limited systems.

teunispeters 2y ago

If it can run X11 on a directly connected display, it's probably not embedded spaces ;). Don't take this comment too seriously, however raspberry pi is definitely at the "more desktop-like" end of the embedding pool. As for systemd, it's an interesting set of tools, and one can if one wishes pick which ones are active in a system, if that system is complex enough to be running Linux (or other compatible OS) to begin with.

m463 2y ago

I think what he's running into is "the unix philosophy". It is basically about small tools, limited in scope, that solve a problem surgically and comprehensively. They work together by mixing and matching to solve the problem. systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing. I don't know, it is sort of like busybox, replacing other system components in a limited Minimum Viable Product way, but more like limited function, not limited resources.

cyberax 2y ago
> I think what he's running into is "the unix philosophy". "Unix philosophy" is throwing crap at the problem, until it kinda-sorta works on the developer's machine.
oguz-ismail 2y ago
> systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing And it's slow. Windows 11 boots in 5 seconds on my laptop, Ubuntu 22 used to take about a minute before I finally made the switch.
- dralley 2y ago
  That almost certainly has little to do with systemd itself, and more to do with which services are enabled on boot.
  - lmm 2y ago
    It's due to systemd in that systemd makes it too fiddly to figure out or change which services are enabled on boot. (I used to know a way to disable certain services on boot under systemd, but it doesn't work any more, and I've reached a state of learned helplessness at this point)
    - vetinari 2y ago
      `systemctl disable` and `systemctl mask` work today just like they worked decade and half ago.

iluvcommunism 2y ago

I like systemd sometimes. Other times it confuses me. Particularly the way cgroups work. The way splicing works. Why oracle mem usage is still uncontrollable when I set user splice restrictions. Idk, I’m still learning it. I don’t like the terminology it uses sometimes too like “snapshot.” When I first read about systemd snapshot I was like oh cool you can snapshot the system. No. Also some esoteric things it supposedly supports don’t work sometimes. I find the idea of systemd targets and fstab having anything to do with each other anathema. Idk.

0xbadcafebee 2y ago

Systemd, Wayland, DBus, and other modern Linux "solutions" are all problematic for the same reason: they are complex monoliths that are extremely difficult to replace, in part or in whole. Systems often won't work without them, because the entire ecosystem has grown dependent on them. You literally can't run a modern desktop without a bunch of "shims" whose sole purpose is to fake being systemd. The best designs, sadly, don't win out. The ones that win out are the ones that serve the interests of the largest players. And the largest players are usually RedHat, Google, and other vendors, who just want to get "their" needs met and move on. They couldn't give a fig about compatibility with some other random tiny project, or an embedded device. And those people working on tiny projects aren't in the room when "standards" are decided for the entire community. There are some hold-outs. Slackware went a long time without it, but next release will unfortunately use systemd. I've run Alpine Linux (systemd-free, so far) as my laptop OS for years. I'll probably switch to another distro that's more user friendly, is easier to contribute to. But it's embarrassing and annoying that we have to pretend to be systemd-compatible just to run a desktop. We should call it "GNU/Systemd/Linux" since it's now one big monolith.

thnkman 2y ago
Where have you heard that Slackware will be using systemd next release? I have been using slackware for years, and this is news to me.
- pwg 2y ago
  There's nothing named "systemd" in the FILE_LIST [1] for Slack64-current (which is what eventually becomes "the next release" at the appropriate time. So it would seem the GP has been taken in by some rumor somewhere. [1] https://ftp.ussg.indiana.edu/linux/slackware/slackware64-current/slackware64/FILE_LIST https://ftp.ussg.indiana.edu/linux/slackware/slackware64-cur...
  - 0xbadcafebee 2y ago
    I thought I saw in the last release a note that the next one would come with systemd. It could be I hallucinated it. Maybe I'm becoming a real LLM!

worthless-trash 2y ago

Excellent plan, go to the other inits, bug bounties are what I do over christmas and init systems and hand rolled init scripts are a great source of security flaws. Taking notes on which distros dont run systemd, for when I want to make some money over christmas.

DaSHacka 2y ago
No need to stop at non-systemd inits! https://security.gentoo.org/glsa https://security.gentoo.org/glsa https://security.archlinux.org/issues/all https://security.archlinux.org/issues/all ^ (ctrl+f systemd) https://ubuntu.com/security/cves?package=systemd&limit=100 https://ubuntu.com/security/cves?package=systemd&limit=100 https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Systemd https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Systemd

typ 2y ago

I have no philosophy issue with systemd. The biggest show-stopper (on embedded Linux) for me is its install size. We have edge devices with only 64M RAM and 128M NAND flash. For an OTA update (via Bluetooth sometimes) only ~20M in total, adding ~5M systemd plus its dependencies doesn't make sense. 5MiB is roughly equivalent to the entire Linux kernel image on those devices. Among the components of systemd, the part that has the lowest added value for our use is dbus. It's designed primarily for desktops and useless for most of our systems, but unfortunately, it is also a hard requirement for systemd.

dzannetandreevn 2y ago

[dead]

nrdvana 2y ago

Try s6 or runit, and mdev from busybox. Mdev is not a "full featured" udev replacement, but that's sort of the point. It does just enough to get the job done and is extensible with scripts. If you want to do the size of embedded where this matters, you also probably want a smaller libc like musl, and then buildroot and busybox are your friends.

richardfey 2y ago

I think author is missing on the Devuan-maintained udev fork.

Jean-Papoulos 2y ago

"I'm using a thing in a way it's not meant to be used and it sucks" Is this even a serious post.

robertlagrant 2y ago
It is a serious post; you've just summarised it appallingly.

m0llusk 2y ago

It is unfortunate that after all this time the basic reasons why systemd was introduced and has been so successful are pervasively misunderstood. > The majority of Linux users are uninterested in the pros and cons of systemd. A small number are violently opposed to it, and a small number are violently opposed to those who are opposed to it. Starting out by framing systemd in terms of opposition deliberately steers attention away from the issues that drive its adoption. The tone is moderate, but this is not good faith argumentation. > I think there’s little argument that the main target for systemd is a general-purpose computer, with a modern, integrated graphical desktop That is one of the targets. The main target is servers that run the web of services typically required to back up modern applications. What if an application depends on a resource server which itself depends on a remote filesystem being mounted and an event service running. Theoretically that can work with System V init, but how tricky is that to put in place and make robust? Administrators for systems vending complex interdependent services are the main target and always have been. > Unfortunately, what makes systemd good for general-purpose desktop applications Get it wrong and then run with it. Failing to understand what is driving adoption of systemd leads to arguments about it that don't really go anywhere. > The more fundamental problem is that the people who most like systemd are distribution managers. Sure, there are administrators who like it, and defend it vigorously; but most end users and administrators don’t really care. Not distribution managers, but system administrators. They are the ones with the complex needs that systemd serves. As such they advocate for it and because of the lack of working alternatives distribution managers switch to it in order to please the largest number of clients. Thinking of systemd in terms of desktop systems and distribution managers means misunderstanding why it is there in the first place.

kj4ips 2y ago

TBH, my biggest gripe is that systemd doesn't play nice with musl or ulibc, I would like to not have to ship a cron, a supervisor, a xinetd equivelant, and a binary just to react to inotify/dnotify events. Though openrc how has a supervisor.

M95D 2y ago

Big systemd on embedded devices is actually A GOOD THING! Imagine what Gentoo, Buildroot/OpenWrt or even Busybox alone could do with so much memory after removing all that bloated OS!