9 ms·

Systemd can be a cause of restrictions on daemons

116 points by zdw 1y ago

zdw 1y ago
I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants. Seeing that podman can run containers as systemd services (see https://codesmash.dev/why-i-ditched-docker-for-podman-and-you-should-too https://codesmash.dev/why-i-ditched-docker-for-podman-and-yo... ), it seems like using containers other than as a distribution mechanism has few advantages, and many disadvantages in terms of dependency updates requiring container rebuilds.
- baby_souffle 1y ago
  > I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants. I see it as _exactly_ the opposite. Podman gives me more or less the same security controls as systemd and the package/delivery problem is solved. Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.
  - JoBrad 1y ago
    > Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot. That would be pretty awesome, actually.
    - o11c 1y ago
      I can already hear the systemd-haters complaining about The One True Unix Way™ is to have tools that only do one thing even if that leaves holes in their functionality. That seems like a `machinectl` task though.
    - speed_spread 1y ago
      That would mean systemd entering package management territory. Now THAT would not be well received.
      - zdw 1y ago
        IMO, docker layering over the OS's built-in package management and update lifecycle in an incompatible ways is far worse than systemd replacing the init system and other service management functionality. Back in the old days (late 90's, early 2k's) as a sysadmin I'd often write scripts to chroot or in other ways isolate services rather than run them as root, so extending the init system to handle those features feels like it's a logical extension, not a incompatible replacement.
      - jeroenhd 1y ago
        systemd-sysupdate already exists. systemd won't run the software repository of course, but with systemd-sysupdate together with some overlay mounts you can get Steam Deck-like ease of use system updates. For software management in R/W environments, there's the podman + systemd combo that'll let you run containers like normal systemd services.
    - jeroenhd 1y ago
      Isn't this literally what podman-systemd does? You don't exactly run a command to pull a container, but just like systemd you place a config file in the right directory, tell podman-systemd to reconfigure itself, and run the service the standard systemd way.
      - baby_souffle 1y ago
        > Isn't this literally what podman-systemd does? That was my point, basically. You have two options: 1) the usual `curl` or `wget` to fetch the binary and the lib(s) and all the work of validating and putting them in place and the like and _then_ you can use a systemd/.service file to set up controls for the bin 2) podman pull and then either ask podman to make a .service file for you or write your own because only one of the two approaches has solved the package/distribution issue, containers are _not_ "less relevant given that systemd can twiddle the same isolation bits"
        zdw 1y ago
        What "validating" does docker/podman pull do that is in excess of a curl of a file? One of the advantages of a real package manager is that it checks signatures on the content that is downloaded. The supply chain on a linux distro's package repos is much harder to break into than typosquatting into a docker registry somewhere.
        baby_souffle 1y ago
        > What "validating" does docker/podman pull do that is in excess of a curl of a file? Every single thing has a sha hash so verifying that I actually downloaded what I meant to download is easy. This gets tedious if I have to `curl https://github.com/someUser/someProject/release/latest.tar.gz https://github.com/someUser/someProject/release/latest.tar.g...` and also get the `tar.gz.sha256` file (if they even publish one ...). Curl supports resuming a partial file (assuming the sending server also does) but it can't "know" ahead of time that the first 1/3 of the file I am downloading has already been fetched because it's also used by $someOtherArtifact. > One of the advantages of a real package manager is that it checks signatures on the content that is downloaded. So does docker/podman. > The supply chain on a linux distro's package repos is much harder to break into than typosquatting into a docker registry somewhere. Perhaps. For every "secure" package repo, I'll show you a much more up-to-date package in AUR/Nix.
  - 9dev 1y ago
    That, and dependency management, no? I’m not going back to installing libwhathaveyou-dev-0.28c1 ever again.
    - gf000 1y ago
      Containers don't solve dependency Management, they just push it a layer up so it's only someone else's problem.
      - trenchpilgrim 1y ago
        That sounds like solving dependency management.
        resize2996 1y ago
        For certain values of 'someone else'
      - curt15 1y ago
        They solve dependency management using isolation. Of course there are other strategies for dependency management.
        gf000 1y ago
        Solved dependency management would involve reproducing your artifacts in the future - isolation is a completely orthogonal thing and container tech just sidesteps this issue.
  - nickysielicki 1y ago
    with podman-systemd/"Quadlet" we're basically there: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
    - trenchpilgrim 1y ago
      I replaced all my home server services with this an uninstalled docker entirely. Been very nice.
  - arianvanp 1y ago
    Literally exists. importctl pull-tar https://example.com/image.tar.gz https://example.com/image.tar.gz && portablectl attach image
    - vlovich123 1y ago
      Did you call him?
  - ndriscoll 1y ago
    nixos kind of does that except better. Usually just set services.foo.enabled to true along with any other config you want. It's also super easy to wrap services in a container if you want, and doing so is kept conceptually separate from dependency management. If you want to make your own systemd service, then referencing a package in `ExecStart` or whatever will make it automatically get pulled in as a dependency.
- miladyincontrol 1y ago
  Container rebuilds are disadvantages? Using mkosi and systemd-nspawn for containers it doesnt really feel that way, still a lot easier to build some distroless app container than to finangle a service to have zero access to other binaries, libraries, or other data entirely. I dont get the distribution "advantage" building em with mkosi but I'd argue it a weakness as far too many are running containers with who-knows-what inside them.
  - abenga 1y ago
    Oddly, "mkosi" is "misfortune" in Swahili.
- oncallthrow 1y ago
  > I feel like Docker and other containerization tools are becoming even less relevant Do you work in the software industry?
  - trenchpilgrim 1y ago
    Docker is absolutely less relevant. My personal machines haven't run Docker for months and my employer is finishing our migration away from Docker in a few months. Containers are as relevant as ever, of course.
    - k__ 1y ago
      What's your alternative? Podman?
      - curt15 1y ago
        For fully serverless (e.g. AWS Lambda) deployments, not even Podman. Just a zip file.
      - trenchpilgrim 1y ago
        containerd for most use cases including Kubernetes clusters, Podman for some cases where you need Pod-like features but Kubernetes is too much overhead. OrbStack for development, using Tilt instead of Docker Compose.
- deleted 1y ago
  [deleted]
- shirro 1y ago
  Docker/podman can be an NPM left-pad solution. Ideally you skip a lot of work by using prebuilt docker files. I think there are times when it is just as easy and safer to use a systemd unit file. Ofcourse it depends a lot on the application. Its a whole lot easier to distribute your own static binary than a closed source app linked to a bunch of ancient libraries.
greatgib 1y ago
Systemd, as usual randomly and suddenly breaking things that worked for decade and for people that asked nothing. Because they know better what you need...
- nickysielicki 1y ago
  And what's your preferred alternative to what's described in the article? Packaging every single service in its own 500mb ubuntu chroot and using docker? Running a local dhcp server and a bridge interface so that you can selectively expose ports? Here's an alternative title for this post: these days, two lines in a systemd service file can easily constrain arbitrary applications to just the files and resources they need, and only those.
  - probably_wrong 1y ago
    My grumpy preferred alternative would be "you're supposed to be an init service. That's not your job".
    - Un1corn 1y ago
      > systemd is a suite of basic building blocks for a Linux system. You can always use a simpler init system if you want
    - nickysielicki 1y ago
      I linked it elsewhere in this thread, but you should really watch this talk, particularly 12:45 through 16:20: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo tl;dr: systemd isn't meant to be an init system, it's meant to manage services, and the alternative world where you don't have a unified system for managing services and events actually sucks.
      - egorfine 1y ago
        > systemd isn't meant to be an init system, it's meant to embrace and extinguish unix philosophy. > the alternative world where [] actually sucks. You should bear in mind that there are plenty of people here on HN who have had the experience of running pre-systemd unixes and they know exactly how much "the alternative world" sucked. Spoiler: not much.
miladyincontrol 1y ago
Systemd haters really are often a masterclass in finding problems with flexible, sanely configurable systems.
- akkartik 1y ago
  When you see a large number of masters spanning diverse skill levels across a population, maybe it's an easy skill to acquire.
- oncallthrow 1y ago
  I genuinely believe that systemd might have the highest “haters” to “benefit-to-humanity” ratio, out of any software project in history.
  - Imustaskforhelp 1y ago
    Hey, now I am interested in more of such softwares overall. Like imagine a list where we can create a form where people can give them and give reasonings or just something. What if I can create a github repo and issues regarding this so that we can discuss about them and I can create a website later if it interests but its a really nice thought experiment. Are we talking more about uh every software including proprietory too? Are we talking about lets say websites too or services (what if we extend it to services like websites or even products outside of software niche into things beyond too, that's interesting too) Another interesting point that comes to my mind might be that cryptocoins might be the lowest inverse to this software project in the sense that I believe that there was very little net positive done to all humanity in general, sure the privacy aspects are nice but still, its not worth having people invest their life savings into it thinking that its going to 100x y'know, I have created a whole article about it being frustated by this idea people think regarding crypto as an investment when it could very well be a crypto"currency" but that's a yap for another day. I really nerded over this and I think I loved it, we need a really good discussion about it :>
    - kelvinjps 1y ago
      My mom and dad live in a country where there is a dictator, and that there has been restrictions to send money there. I'm happy that I can send crypto usdc over Solana and then the cost is basically a few cents to my family
dextercd 1y ago
You can use systemd-run with --shell (or a subset of options enabled by --shell) and -p to specify service properties to run commands interactively in a similar environment as your service. This can help troubleshoot issues and makes experimenting with systemd options faster. I think there's been some talk about adding a built-in way for systemd-run to copy settings out of a .service file, but it doesn't exist yet. I've written Perl/Python scripts to do this for me. They're not really aimed at working with arbitrary services, but it should be possible to adapt to different scenarios. https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd8dd49 https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd... There are some gotchas I ran into. For example, with RuntimeDirectory: systemd deletes the directory once the process exits, even if there's still another process running with the same RuntimeDirectory value set.
- matheusmoreira 1y ago
  I use systemd-run very often to impose CPU usage limits on software. Awesome feature.
  - Imustaskforhelp 1y ago
    woah that's actually awesome. I feel like adding uh storage usage limits could also be easy as well. But the one thing that I always wonder is about (virtualization?) in the sense of something like docker just for containerizing or some sort of way of running them in some sort of sandbox without much performance issues or something, I am kinda interested in knowing what might be the best way of doing so (is podman the right way or some other way like bubblewrap?) Edit: just discovered in the comment below the (parents parents?)comment that there is systemd isolation too, that sounds very interesting and the first time I personally heard of it hmm
    - nullpoint420 1y ago
      Kubernetes is also this but for the cloud. CPU/Mem/Storage limits
    - ChocolateGod 1y ago
Zardoz84 1y ago
The private /tmp strike us, when update to Debían 12 servers and find that a batch process cannot access the same temporal files that our web application. Luckily, it's very easy to fix, adding an extra systems file to disable that feature on the Tomcat service.
- egorfine 1y ago
  Problem is, those new systemd "features" [1] have been shoved down our throats with little to no warning. [1] https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189 https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-ac...
serbuvlad 1y ago
> One of the traditional rites of passage for Linux system administrators is having a daemon not work in the normal system configuration (eg, when you boot the system) but work when you manually run it as root. I've don't remember the last time I run a daemon by hand (that I wasn't developed it myself). I always just run the systemd unit via systemctl and debug that. > A standard thing I do when troubleshooting a chain of programs executing programs executing programs is to shim in diagnostics that dump information to /tmp. This seems like a very esoteric case in the days of structured logging and log levels. > A mailer usually can't really tell the difference between 'no one has .forward files' and 'I'm mysteriously not able to see people's home directories to find .forward files in them' Obviously a daemon that should access files in people's home directories shouldn't have ProtectHome=true. It's the responsibility of the daemon developer or the package maintainer to set appropriate flags based on what the daemon does. Someone had to explicitly write "ProtectHome=true". It's not the default, and it doesn't just appear in the service file. When in doubt don't set security options at all, instead of shipping a broken daemon that you don't understand why it doesn't work. Note: please base your daemon on D-Bus or a socket in /run and not on reading arbitrary files from my home directory. I also don't understand the larger perspective? Should we not make our daemon run in more secure environments?
- egorfine 1y ago
  > in the days of structured logging and Are those days in the same room we are? Because last I have checked, logging is generally implemented as a stream of text messages with a very few bits of structured metadata. Like, almost in every piece of software in existence. > log levels. Those exist for decades and were invented long before anything that could be considered "structured logging".
- egorfine 1y ago
  > Should we not make our daemon run in more secure environments? I want systemd to get out of my lawn and stop adding features. It's not the job of /sbin/init to setup specifics of the secure environment. I want my init to launch daemons and that's it.
amaccuish 1y ago
Old Man Yells at Cloud.
- ranger207 1y ago
  Nah, this blog is neutral about it at worst. He publishes daily, and it's usually tips or notes on how things work, especially if it's different from the "traditional" way of doing things. One of my favorite blogs for letting me know how things work nowadays
amelius 1y ago
I'm ok with it as long as it doesn't cause __any__ confusion whatsoever.
nine_k 1y ago
What makes me scratch my hand is why the failed access violations are not easy to show and log. A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally, this also should be made explicit, either in the service code, or in its configuration. There should be an strace-like tool that would collect a log of such "access denied" erros for troubleshooting. Even better, each service should run in its own process group, and tracing could be switched on / off for a particular process group.
- cesarb 1y ago
  > A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally It's normally recommended to attempt the access and handle the denial, instead of doing two separate steps (checking for access and doing the access); the later can lead to security issues (https://en.wikipedia.org/wiki/TOCTOU https://en.wikipedia.org/wiki/TOCTOU).
  - nine_k 1y ago
    Yes, this is the explicit attempt of access which should be logged by the service.
jmclnx 1y ago
>Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features. https://cscie2x.dce.harvard.edu/hw/ch01s06.html https://cscie2x.dce.harvard.edu/hw/ch01s06.html systemd seems to violate this at every turn, but sadly they are not the only ones to do that in Linux. I wonder at what point we should start considering Linux to be its own thing instead of "UN*X like".
- r14c 1y ago
  I know people like to complain, but these comments are not really based in reality. systemd is really a suite of applications that work together to provide init, service management, network management, &c. there's no single giant program that does all this. each component has a specific focused task. you could break this out into individually named projects, but nobody complains about coreutils enabling dozens of different workflows.
  - egorfine 1y ago
    > there's no single giant program that does all this systemd crown does everything for that to be technically legally right while also making sure that and modern Linux distro is unusable without a bunch of systemd-* projects.
    - r14c 1y ago
      If you don't want systemd, pick a distro that doesn't use systemd. You'll be a lot happier doing that than fighting a distro that has chosen to adopt a set of tools that you dislike.
      - egorfine 1y ago
        At this point we're not talking about distros w/o systemd as there are none left speaking from a practical perspective. We should be discussing management of the metastases, because systemd people still exist and they still want attention and suffering and thus they are still looking for existing AND working tools to destroy.
OutOfHere 1y ago
systemd restrictions on daemons are an excellent feature for daemon security, especially for publicly accessed services. The restrictions are fully configurable and can be removed, so there is nothing to lose if you can control the service definition file. Moreover, at least the GPT LLM is well versed in them, making the unit file easy to write or update.
prima-facie 1y ago
Systemd hardening is great, but each service needs its own bespoke config and that takes a bit of time and trial & error. Here's the override I've been using for Jellyfin: https://gist.github.com/radupotop/61d59052ff0a81cc5a32c92b3bfaa1d6 https://gist.github.com/radupotop/61d59052ff0a81cc5a32c92b3b... Some references: - https://docs.arbitrary.ch/security/systemd.html https://docs.arbitrary.ch/security/systemd.html - https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db04 https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db...
NewJazz 1y ago
I don't understand how people consider this article "systemd hate". The article is informative. Even a bit bland when it comes to opinion on the matter of systemd as a whole. The article is literally just saying "if you write services, complain loudly and with context about permission errors" and "if you use systemD with hardening enabled, consider it alongside discretionary access control and mandatory access control when troubleshooting permissions errors".
godelski 1y ago
Make sure to do systemctl edit foo.service The reason for this is that it creates an override file rather than edits the systemd file directly. This means you'll keep your changes even if the systemd file changes in an upgrade. Obviously also helps you roll back and make sure you got things right. Another side benefit is that you can place these on github. Also, pretty much every service should use `PrivateTmp=yes` and this is surprisingly uncommon. Then run systemd-analyze security foo.service It'll give you a nice little rundown of what is restricted and what isn't. There's a short description of what each of the capabilities are, but I also like this gist[0]. The docs kinda suck (across systemd) but play around and it starts to make sense. This stuff is surprisingly quite powerful and it's caused me to go on a systemd binge for the last year. You can restrict access to basically anything, effectively giving you near container like restrictions. You can restrict paths (and permissions to those paths), executables, what conditions will cause the service to start/stop, how much CPU and memory usage a service can grab, and all sorts of things. That last one can be really helpful when you have a service that is being too greedy, giving you effectively an extremely lightweight container. If you want to go further, then look into systemd-nspawn and systemd-vmspawn. This will give you a "chroot on steroids" (lightweight container, which can be used in place of things like docker) and a full fledged VM, respectively. Both of these have two complementary commands: machinectl and importctl. `machinectl` will allow you to make these things act just like services, even allowing you to define conditions in which they can autoactivate. `importctl` gives you the ability to download, import, and export these machines as images. You can use importctl to get an image from an arbitrary url, so if a project decided to provide this in addition to (or as a replacement to) a docker image, you could have a pretty similar streamlined process. I'm no expert, but I've not run across a case where you can't use systemd as a full on replacement for docker. While it's more annoying to make these images I've found that I can get better control over them and feel a lot less like I'm fighting the system. A big advantage too is that these can run with fewer resources than docker, but this really depends on tuning. The more locked down the slower it will be, so this is only a "can" not "will" (IIRC defaults on vmspawn are slower[1]) Finally, note that all these have a `--user` flag, so you don't always need to use sudo. That itself gives a big advantage because you're automatically limiting capabilities by running it as a user service. No more fucking around with groups and all that. Honestly, the biggest downside of systemd is the lack of documentation and the lack of wider adoption and examples to pull from. But that's a solvable problem and the exact kinda thing HN can help solve (this comment is even doing some of that as well as some others in this thread). Systemd is far from perfect and there's without a doubt a good learning curve (i.e. lack of docs), but IME it's been worth the effort. [0] https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db04 https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db... [1] https://github.com/systemd/systemd/issues/18370 https://github.com/systemd/systemd/issues/18370
egorfine 1y ago
> The bad news [...] with the assumption that DNS resolution is being done via systemd-resolved This is incredible, really. systemd-resolved is easily the worst piece that came out of systemd crowd. I see systemd-resolved as a transparent film that protects the shiny parts of new devices: something that is meant to be removed immediately after unboxing. This, and timesyncd. Now, it's no wonder systemd crowd now want this thing mandatory. They truly want to be noticed and the best way to be noticed is to make someone suffer while virtue signaling righteousness.
- koverstreet 1y ago
  If something's garbage you can just call it garbage, no need to wrap it up in virtue signaling and culture wars. But please do tell us why it's garbage
  - egorfine 1y ago
    It is as costly to call systemd-* garbage on HN as it is to eliminate systemd-* projects from major distros.
    - koverstreet 1y ago
      Well, if you don't include the "why", to you aren't saying anything everyone hasn't heard a thousand times before. The "why" is important.
      - egorfine 1y ago
        That is true. I have no new arguments. Today I have learned about PrivateTmp and added that thing to the evergrowing list of things to get rid of on fresh Debian instances. Yes, I did think for ten seconds and I am pretty sure I don't want that at all at any of the servers I am managing. As for systemd-resolved, it was broken beyond repairability many years ago right at the start. At first I have tried to read manuals and play along with it. All I needed was OS using my resolvers only. Simple as "nameserver 1.1.1.1" in resolv.conf. Unfortunately, no matter how much I read the documentation, I couldn't make it do that one simple job. Oh, and manually editing resolve.conf was not an option as well because it randomly overwrote it. On a server with a static ip, mind you, no dhcp. Maybe my memory is serving me wrong but I think systemd-resolved was so bad that even +x on /etc/resolv.conf did not work, it would still overwrite it. Incredible piece of shit, just incredible. I am now removing it from all instances by default for years and I refuse to learn whether they made good or not. Clearly systemd-resolved provides negative value to the world.
egorfine 1y ago
By that point it should be pretty clear that anyone who opposes the one and only Right Way™ of systemd - are old and don't want to learn new ways. Obviously systemd is perfect and their philosophy is the best and not to be criticized. /s (there goes my karma)

116 points by zdw 1y ago

zdw 1y ago

I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants. Seeing that podman can run containers as systemd services (see https://codesmash.dev/why-i-ditched-docker-for-podman-and-you-should-too https://codesmash.dev/why-i-ditched-docker-for-podman-and-yo... ), it seems like using containers other than as a distribution mechanism has few advantages, and many disadvantages in terms of dependency updates requiring container rebuilds.

baby_souffle 1y ago
> I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants. I see it as _exactly_ the opposite. Podman gives me more or less the same security controls as systemd and the package/delivery problem is solved. Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.
- JoBrad 1y ago
  > Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot. That would be pretty awesome, actually.
  - o11c 1y ago
    I can already hear the systemd-haters complaining about The One True Unix Way™ is to have tools that only do one thing even if that leaves holes in their functionality. That seems like a `machinectl` task though.
  - speed_spread 1y ago
    That would mean systemd entering package management territory. Now THAT would not be well received.
    - zdw 1y ago
      IMO, docker layering over the OS's built-in package management and update lifecycle in an incompatible ways is far worse than systemd replacing the init system and other service management functionality. Back in the old days (late 90's, early 2k's) as a sysadmin I'd often write scripts to chroot or in other ways isolate services rather than run them as root, so extending the init system to handle those features feels like it's a logical extension, not a incompatible replacement.
    - jeroenhd 1y ago
      systemd-sysupdate already exists. systemd won't run the software repository of course, but with systemd-sysupdate together with some overlay mounts you can get Steam Deck-like ease of use system updates. For software management in R/W environments, there's the podman + systemd combo that'll let you run containers like normal systemd services.
  - jeroenhd 1y ago
    Isn't this literally what podman-systemd does? You don't exactly run a command to pull a container, but just like systemd you place a config file in the right directory, tell podman-systemd to reconfigure itself, and run the service the standard systemd way.
    - baby_souffle 1y ago
      > Isn't this literally what podman-systemd does? That was my point, basically. You have two options: 1) the usual `curl` or `wget` to fetch the binary and the lib(s) and all the work of validating and putting them in place and the like and _then_ you can use a systemd/.service file to set up controls for the bin 2) podman pull and then either ask podman to make a .service file for you or write your own because only one of the two approaches has solved the package/distribution issue, containers are _not_ "less relevant given that systemd can twiddle the same isolation bits"
      - zdw 1y ago
        What "validating" does docker/podman pull do that is in excess of a curl of a file? One of the advantages of a real package manager is that it checks signatures on the content that is downloaded. The supply chain on a linux distro's package repos is much harder to break into than typosquatting into a docker registry somewhere.
        baby_souffle 1y ago
        > What "validating" does docker/podman pull do that is in excess of a curl of a file? Every single thing has a sha hash so verifying that I actually downloaded what I meant to download is easy. This gets tedious if I have to `curl https://github.com/someUser/someProject/release/latest.tar.gz https://github.com/someUser/someProject/release/latest.tar.g...` and also get the `tar.gz.sha256` file (if they even publish one ...). Curl supports resuming a partial file (assuming the sending server also does) but it can't "know" ahead of time that the first 1/3 of the file I am downloading has already been fetched because it's also used by $someOtherArtifact. > One of the advantages of a real package manager is that it checks signatures on the content that is downloaded. So does docker/podman. > The supply chain on a linux distro's package repos is much harder to break into than typosquatting into a docker registry somewhere. Perhaps. For every "secure" package repo, I'll show you a much more up-to-date package in AUR/Nix.
- 9dev 1y ago
  That, and dependency management, no? I’m not going back to installing libwhathaveyou-dev-0.28c1 ever again.
  - gf000 1y ago
    Containers don't solve dependency Management, they just push it a layer up so it's only someone else's problem.
    - trenchpilgrim 1y ago
      That sounds like solving dependency management.
      - resize2996 1y ago
        For certain values of 'someone else'
    - curt15 1y ago
      They solve dependency management using isolation. Of course there are other strategies for dependency management.
      - gf000 1y ago
        Solved dependency management would involve reproducing your artifacts in the future - isolation is a completely orthogonal thing and container tech just sidesteps this issue.
- nickysielicki 1y ago
  with podman-systemd/"Quadlet" we're basically there: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
  - trenchpilgrim 1y ago
    I replaced all my home server services with this an uninstalled docker entirely. Been very nice.
- arianvanp 1y ago
  Literally exists. importctl pull-tar https://example.com/image.tar.gz https://example.com/image.tar.gz && portablectl attach image
  - vlovich123 1y ago
    Did you call him?
- ndriscoll 1y ago
  nixos kind of does that except better. Usually just set services.foo.enabled to true along with any other config you want. It's also super easy to wrap services in a container if you want, and doing so is kept conceptually separate from dependency management. If you want to make your own systemd service, then referencing a package in `ExecStart` or whatever will make it automatically get pulled in as a dependency.
miladyincontrol 1y ago
Container rebuilds are disadvantages? Using mkosi and systemd-nspawn for containers it doesnt really feel that way, still a lot easier to build some distroless app container than to finangle a service to have zero access to other binaries, libraries, or other data entirely. I dont get the distribution "advantage" building em with mkosi but I'd argue it a weakness as far too many are running containers with who-knows-what inside them.
- abenga 1y ago
  Oddly, "mkosi" is "misfortune" in Swahili.
oncallthrow 1y ago
> I feel like Docker and other containerization tools are becoming even less relevant Do you work in the software industry?
- trenchpilgrim 1y ago
  Docker is absolutely less relevant. My personal machines haven't run Docker for months and my employer is finishing our migration away from Docker in a few months. Containers are as relevant as ever, of course.
  - k__ 1y ago
    What's your alternative? Podman?
    - curt15 1y ago
      For fully serverless (e.g. AWS Lambda) deployments, not even Podman. Just a zip file.
    - trenchpilgrim 1y ago
      containerd for most use cases including Kubernetes clusters, Podman for some cases where you need Pod-like features but Kubernetes is too much overhead. OrbStack for development, using Tilt instead of Docker Compose.
deleted 1y ago
[deleted]
shirro 1y ago
Docker/podman can be an NPM left-pad solution. Ideally you skip a lot of work by using prebuilt docker files. I think there are times when it is just as easy and safer to use a systemd unit file. Ofcourse it depends a lot on the application. Its a whole lot easier to distribute your own static binary than a closed source app linked to a bunch of ancient libraries.

greatgib 1y ago

Systemd, as usual randomly and suddenly breaking things that worked for decade and for people that asked nothing. Because they know better what you need...

nickysielicki 1y ago
And what's your preferred alternative to what's described in the article? Packaging every single service in its own 500mb ubuntu chroot and using docker? Running a local dhcp server and a bridge interface so that you can selectively expose ports? Here's an alternative title for this post: these days, two lines in a systemd service file can easily constrain arbitrary applications to just the files and resources they need, and only those.
- probably_wrong 1y ago
  My grumpy preferred alternative would be "you're supposed to be an init service. That's not your job".
  - Un1corn 1y ago
    > systemd is a suite of basic building blocks for a Linux system. You can always use a simpler init system if you want
  - nickysielicki 1y ago
    I linked it elsewhere in this thread, but you should really watch this talk, particularly 12:45 through 16:20: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo tl;dr: systemd isn't meant to be an init system, it's meant to manage services, and the alternative world where you don't have a unified system for managing services and events actually sucks.
    - egorfine 1y ago
      > systemd isn't meant to be an init system, it's meant to embrace and extinguish unix philosophy. > the alternative world where [] actually sucks. You should bear in mind that there are plenty of people here on HN who have had the experience of running pre-systemd unixes and they know exactly how much "the alternative world" sucked. Spoiler: not much.

miladyincontrol 1y ago

Systemd haters really are often a masterclass in finding problems with flexible, sanely configurable systems.

akkartik 1y ago
When you see a large number of masters spanning diverse skill levels across a population, maybe it's an easy skill to acquire.
oncallthrow 1y ago
I genuinely believe that systemd might have the highest “haters” to “benefit-to-humanity” ratio, out of any software project in history.
- Imustaskforhelp 1y ago
  Hey, now I am interested in more of such softwares overall. Like imagine a list where we can create a form where people can give them and give reasonings or just something. What if I can create a github repo and issues regarding this so that we can discuss about them and I can create a website later if it interests but its a really nice thought experiment. Are we talking more about uh every software including proprietory too? Are we talking about lets say websites too or services (what if we extend it to services like websites or even products outside of software niche into things beyond too, that's interesting too) Another interesting point that comes to my mind might be that cryptocoins might be the lowest inverse to this software project in the sense that I believe that there was very little net positive done to all humanity in general, sure the privacy aspects are nice but still, its not worth having people invest their life savings into it thinking that its going to 100x y'know, I have created a whole article about it being frustated by this idea people think regarding crypto as an investment when it could very well be a crypto"currency" but that's a yap for another day. I really nerded over this and I think I loved it, we need a really good discussion about it :>
  - kelvinjps 1y ago
    My mom and dad live in a country where there is a dictator, and that there has been restrictions to send money there. I'm happy that I can send crypto usdc over Solana and then the cost is basically a few cents to my family

dextercd 1y ago

You can use systemd-run with --shell (or a subset of options enabled by --shell) and -p to specify service properties to run commands interactively in a similar environment as your service. This can help troubleshoot issues and makes experimenting with systemd options faster. I think there's been some talk about adding a built-in way for systemd-run to copy settings out of a .service file, but it doesn't exist yet. I've written Perl/Python scripts to do this for me. They're not really aimed at working with arbitrary services, but it should be possible to adapt to different scenarios. https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd8dd49 https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd... There are some gotchas I ran into. For example, with RuntimeDirectory: systemd deletes the directory once the process exits, even if there's still another process running with the same RuntimeDirectory value set.

matheusmoreira 1y ago
I use systemd-run very often to impose CPU usage limits on software. Awesome feature.
- Imustaskforhelp 1y ago
  woah that's actually awesome. I feel like adding uh storage usage limits could also be easy as well. But the one thing that I always wonder is about (virtualization?) in the sense of something like docker just for containerizing or some sort of way of running them in some sort of sandbox without much performance issues or something, I am kinda interested in knowing what might be the best way of doing so (is podman the right way or some other way like bubblewrap?) Edit: just discovered in the comment below the (parents parents?)comment that there is systemd isolation too, that sounds very interesting and the first time I personally heard of it hmm
  - nullpoint420 1y ago
    Kubernetes is also this but for the cloud. CPU/Mem/Storage limits
  - ChocolateGod 1y ago

Zardoz84 1y ago

The private /tmp strike us, when update to Debían 12 servers and find that a batch process cannot access the same temporal files that our web application. Luckily, it's very easy to fix, adding an extra systems file to disable that feature on the Tomcat service.

egorfine 1y ago
Problem is, those new systemd "features" [1] have been shoved down our throats with little to no warning. [1] https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189 https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-ac...

serbuvlad 1y ago

> One of the traditional rites of passage for Linux system administrators is having a daemon not work in the normal system configuration (eg, when you boot the system) but work when you manually run it as root. I've don't remember the last time I run a daemon by hand (that I wasn't developed it myself). I always just run the systemd unit via systemctl and debug that. > A standard thing I do when troubleshooting a chain of programs executing programs executing programs is to shim in diagnostics that dump information to /tmp. This seems like a very esoteric case in the days of structured logging and log levels. > A mailer usually can't really tell the difference between 'no one has .forward files' and 'I'm mysteriously not able to see people's home directories to find .forward files in them' Obviously a daemon that should access files in people's home directories shouldn't have ProtectHome=true. It's the responsibility of the daemon developer or the package maintainer to set appropriate flags based on what the daemon does. Someone had to explicitly write "ProtectHome=true". It's not the default, and it doesn't just appear in the service file. When in doubt don't set security options at all, instead of shipping a broken daemon that you don't understand why it doesn't work. Note: please base your daemon on D-Bus or a socket in /run and not on reading arbitrary files from my home directory. I also don't understand the larger perspective? Should we not make our daemon run in more secure environments?

egorfine 1y ago
> in the days of structured logging and Are those days in the same room we are? Because last I have checked, logging is generally implemented as a stream of text messages with a very few bits of structured metadata. Like, almost in every piece of software in existence. > log levels. Those exist for decades and were invented long before anything that could be considered "structured logging".
egorfine 1y ago
> Should we not make our daemon run in more secure environments? I want systemd to get out of my lawn and stop adding features. It's not the job of /sbin/init to setup specifics of the secure environment. I want my init to launch daemons and that's it.

amaccuish 1y ago

Old Man Yells at Cloud.

ranger207 1y ago
Nah, this blog is neutral about it at worst. He publishes daily, and it's usually tips or notes on how things work, especially if it's different from the "traditional" way of doing things. One of my favorite blogs for letting me know how things work nowadays

amelius 1y ago

I'm ok with it as long as it doesn't cause __any__ confusion whatsoever.

nine_k 1y ago

What makes me scratch my hand is why the failed access violations are not easy to show and log. A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally, this also should be made explicit, either in the service code, or in its configuration. There should be an strace-like tool that would collect a log of such "access denied" erros for troubleshooting. Even better, each service should run in its own process group, and tracing could be switched on / off for a particular process group.

cesarb 1y ago
> A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally It's normally recommended to attempt the access and handle the denial, instead of doing two separate steps (checking for access and doing the access); the later can lead to security issues (https://en.wikipedia.org/wiki/TOCTOU https://en.wikipedia.org/wiki/TOCTOU).
- nine_k 1y ago
  Yes, this is the explicit attempt of access which should be logged by the service.

jmclnx 1y ago

>Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features. https://cscie2x.dce.harvard.edu/hw/ch01s06.html https://cscie2x.dce.harvard.edu/hw/ch01s06.html systemd seems to violate this at every turn, but sadly they are not the only ones to do that in Linux. I wonder at what point we should start considering Linux to be its own thing instead of "UN*X like".

r14c 1y ago
I know people like to complain, but these comments are not really based in reality. systemd is really a suite of applications that work together to provide init, service management, network management, &c. there's no single giant program that does all this. each component has a specific focused task. you could break this out into individually named projects, but nobody complains about coreutils enabling dozens of different workflows.
- egorfine 1y ago
  > there's no single giant program that does all this systemd crown does everything for that to be technically legally right while also making sure that and modern Linux distro is unusable without a bunch of systemd-* projects.
  - r14c 1y ago
    If you don't want systemd, pick a distro that doesn't use systemd. You'll be a lot happier doing that than fighting a distro that has chosen to adopt a set of tools that you dislike.
    - egorfine 1y ago
      At this point we're not talking about distros w/o systemd as there are none left speaking from a practical perspective. We should be discussing management of the metastases, because systemd people still exist and they still want attention and suffering and thus they are still looking for existing AND working tools to destroy.

OutOfHere 1y ago

systemd restrictions on daemons are an excellent feature for daemon security, especially for publicly accessed services. The restrictions are fully configurable and can be removed, so there is nothing to lose if you can control the service definition file. Moreover, at least the GPT LLM is well versed in them, making the unit file easy to write or update.

prima-facie 1y ago

Systemd hardening is great, but each service needs its own bespoke config and that takes a bit of time and trial & error. Here's the override I've been using for Jellyfin: https://gist.github.com/radupotop/61d59052ff0a81cc5a32c92b3bfaa1d6 https://gist.github.com/radupotop/61d59052ff0a81cc5a32c92b3b... Some references: - https://docs.arbitrary.ch/security/systemd.html https://docs.arbitrary.ch/security/systemd.html - https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db04 https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db...

NewJazz 1y ago

I don't understand how people consider this article "systemd hate". The article is informative. Even a bit bland when it comes to opinion on the matter of systemd as a whole. The article is literally just saying "if you write services, complain loudly and with context about permission errors" and "if you use systemD with hardening enabled, consider it alongside discretionary access control and mandatory access control when troubleshooting permissions errors".

godelski 1y ago

Make sure to do systemctl edit foo.service The reason for this is that it creates an override file rather than edits the systemd file directly. This means you'll keep your changes even if the systemd file changes in an upgrade. Obviously also helps you roll back and make sure you got things right. Another side benefit is that you can place these on github. Also, pretty much every service should use `PrivateTmp=yes` and this is surprisingly uncommon. Then run systemd-analyze security foo.service It'll give you a nice little rundown of what is restricted and what isn't. There's a short description of what each of the capabilities are, but I also like this gist[0]. The docs kinda suck (across systemd) but play around and it starts to make sense. This stuff is surprisingly quite powerful and it's caused me to go on a systemd binge for the last year. You can restrict access to basically anything, effectively giving you near container like restrictions. You can restrict paths (and permissions to those paths), executables, what conditions will cause the service to start/stop, how much CPU and memory usage a service can grab, and all sorts of things. That last one can be really helpful when you have a service that is being too greedy, giving you effectively an extremely lightweight container. If you want to go further, then look into systemd-nspawn and systemd-vmspawn. This will give you a "chroot on steroids" (lightweight container, which can be used in place of things like docker) and a full fledged VM, respectively. Both of these have two complementary commands: machinectl and importctl. `machinectl` will allow you to make these things act just like services, even allowing you to define conditions in which they can autoactivate. `importctl` gives you the ability to download, import, and export these machines as images. You can use importctl to get an image from an arbitrary url, so if a project decided to provide this in addition to (or as a replacement to) a docker image, you could have a pretty similar streamlined process. I'm no expert, but I've not run across a case where you can't use systemd as a full on replacement for docker. While it's more annoying to make these images I've found that I can get better control over them and feel a lot less like I'm fighting the system. A big advantage too is that these can run with fewer resources than docker, but this really depends on tuning. The more locked down the slower it will be, so this is only a "can" not "will" (IIRC defaults on vmspawn are slower[1]) Finally, note that all these have a `--user` flag, so you don't always need to use sudo. That itself gives a big advantage because you're automatically limiting capabilities by running it as a user service. No more fucking around with groups and all that. Honestly, the biggest downside of systemd is the lack of documentation and the lack of wider adoption and examples to pull from. But that's a solvable problem and the exact kinda thing HN can help solve (this comment is even doing some of that as well as some others in this thread). Systemd is far from perfect and there's without a doubt a good learning curve (i.e. lack of docs), but IME it's been worth the effort. [0] https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db04 https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db... [1] https://github.com/systemd/systemd/issues/18370 https://github.com/systemd/systemd/issues/18370

egorfine 1y ago

> The bad news [...] with the assumption that DNS resolution is being done via systemd-resolved This is incredible, really. systemd-resolved is easily the worst piece that came out of systemd crowd. I see systemd-resolved as a transparent film that protects the shiny parts of new devices: something that is meant to be removed immediately after unboxing. This, and timesyncd. Now, it's no wonder systemd crowd now want this thing mandatory. They truly want to be noticed and the best way to be noticed is to make someone suffer while virtue signaling righteousness.

koverstreet 1y ago
If something's garbage you can just call it garbage, no need to wrap it up in virtue signaling and culture wars. But please do tell us why it's garbage
- egorfine 1y ago
  It is as costly to call systemd-* garbage on HN as it is to eliminate systemd-* projects from major distros.
  - koverstreet 1y ago
    Well, if you don't include the "why", to you aren't saying anything everyone hasn't heard a thousand times before. The "why" is important.
    - egorfine 1y ago
      That is true. I have no new arguments. Today I have learned about PrivateTmp and added that thing to the evergrowing list of things to get rid of on fresh Debian instances. Yes, I did think for ten seconds and I am pretty sure I don't want that at all at any of the servers I am managing. As for systemd-resolved, it was broken beyond repairability many years ago right at the start. At first I have tried to read manuals and play along with it. All I needed was OS using my resolvers only. Simple as "nameserver 1.1.1.1" in resolv.conf. Unfortunately, no matter how much I read the documentation, I couldn't make it do that one simple job. Oh, and manually editing resolve.conf was not an option as well because it randomly overwrote it. On a server with a static ip, mind you, no dhcp. Maybe my memory is serving me wrong but I think systemd-resolved was so bad that even +x on /etc/resolv.conf did not work, it would still overwrite it. Incredible piece of shit, just incredible. I am now removing it from all instances by default for years and I refuse to learn whether they made good or not. Clearly systemd-resolved provides negative value to the world.

egorfine 1y ago

By that point it should be pretty clear that anyone who opposes the one and only Right Way™ of systemd - are old and don't want to learn new ways. Obviously systemd is perfect and their philosophy is the best and not to be criticized. /s (there goes my karma)