5 ms·

Things that are terrible: logging. Why did we have to screw this up? Just writing stuff to a file was fine. Things that are good: simple syntax, easy to use ut

Things that are terrible: logging. Why did we have to screw this up? Just writing stuff to a file was fine. Things that are good: simple syntax, easy to use utils Hidden gems: cgroups, file systems containment, etc. Systemd could completely eliminate the need for stuff like docker by totally isolating a process.

porsager 6y ago
That's interesting. Do you have a link to more info about isolating a process like that?
- comboy 6y ago
  Arch wiki is usually a good place to learn something in a succinct way (I've never even used Arch): https://wiki.archlinux.org/title/cgroups https://wiki.archlinux.org/title/cgroups
moshmosh 6y ago
The nice thing about docker is that you can use the same tooling on lots of operating systems, not just that it wraps some Linux functionality. Sure, it may involve using a VM, depending on the system, but I still get to just run "docker" and "docker-compose" and such, as if it were native. A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. Really, though, if I care about isolation and don't want to use docker, I'm gonna reach for FreeBSD + jails anyway, and I don't think systemd's gonna change that for me any time soon. If I want containers with better tooling outside the target OS, I'll use docker. Systemd falls in an awkward middle that I don't have much interest in.
- vially 6y ago
  > A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. But that's exactly what happens with docker anyway (e.g.: Docker is still running in the Linux VM when running it from a different operating system). And considering the widespread systemd adoption throughout the Linux ecosystem, it's far more likely for systemd to be already installed in the VM than docker anyway.
  - moshmosh 6y ago
    Docker on Linux runs in a VM? No. Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though. Systemd-docker-replacement: works only on systemd-Linux. Tooling for any other arrangement, including non-systemd linux, left as an exercise to the user. Docker: works on Linux—systemd or otherwise. Has tooling to making working on macOS or Windows non-terrible and lets you use (mostly) the same commands as you would on Linux. I'm saying that cross-platform tooling is what makes Docker Docker. Systemd can't replace it without replacing the tooling, even if it can do the same thing only on systemd-Linux.
    - vially 5y ago
      > Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though. Would it? There's no reason this hypothetical systemd-docker-replacement couldn't be architected in a way that would not have a runtime dependency on systemd-the-init-system. In any case, I'm not even sure if systemd wants to be a docker replacement (although it does seem to pick up more and more container features lately). But there's definitely some overlap between the two projects (in particular around process/service management) but podman is a much more direct competitor to docker than systemd.
comboy 6y ago
It's just status quo. How would you feel about introducing having to log things into file after all you needed was stdout? Btw, you still can do that nothing is stopping you, but I don't think each and every service needs to decide (and let user configure) where to put its logs, how to separate them (maybe by errors, maybe by users/domains) and how to rotate them. I'm still not a big fan of systemd but it does make some things easier (but why wouldn't they think for a moment how easy "systemctl restoptart foo" will be to type, tab complete and issue different commands one after another)
dale_glass 6y ago
The logging is amazing. Transparent compression, easy search, output in JSON if you want to parse logs, timestamps with microseconds if you need them. No, writing stuff to a file was not fine. It's only fine if all you do is to 'tail' and 'grep' it, but any time you want to do something more advanced, it's a bloody mess. Why parse a text date, when it started its life as an unix timestamp? Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log that you could just search for? Why deal with parsing while the log may rotate or truncate under you, when the log system could be making your life easier? I wrote a whole bunch of log parsing stuff in my day, and so I have no nostalgia whatsoever regarding this "just write stuff to a file" approach.
- Arch-TK 6y ago
  > The logging is amazing. Amazing for you maybe. If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration). Meanwhile cat or tail -f of a log file is almost instant somehow. > Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log I mean, this is a problem which doesn't require systemd to solve, you just need a better log format. Someone could have standardised a line based logging format with fields and then someone else could have written libraries to write it and libraries to parse it. > Why deal with parsing while the log may rotate or truncate under you That's a log daemon problem that I haven't found to exist with any log daemon I have used.
  - dale_glass 6y ago
    > If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration). That's because you're asking for a log of a service since logging began, which may well have been months worth of logging. You're saying "give me everything that happened ever to this service in a pager, then scroll down to the bottom". If you want tail -f, use a -f: 'journalctl -f -u <service>'. That's going to be a lot faster. You can also use '-b' to make it return a log of what happened since boot (worth looking into the manual for the details of that one), or specify a start date with -S, like '-S yesterday'. > Someone could have standardised a line based logging format And what were they waiting for? In the end, systemd did it, but it's a pretty recent invention. There was plenty time to get there first. > That's a log daemon problem that I haven't found to exist with any log daemon I have used. Because log daemons classically don't care about this at all -- it's a problem for whatever reads the files that the log daemon generates afterwards.
ATsch 6y ago
> Just writing stuff to a file was fine. Haha, I wish. Then that file becomes too large. So you want to truncate the start. However, removing bytes from the start of an open file is a bad idea. So you open a new log file every day instead. You really want to compress the old log files too, to save space. So now you can no longer just grep the logs as before. You're now also duplicating code and configuration for every daemon. So you write a daemon called `logrotate` that does this centrally instead. Into which you then copy the log paths configured for every program on your system and how long the logs should get kept. However programs don't much like it when you move the log files out from under them. So you patch every program respond to a signal by checking whether it's log files have been moved. Which then breaks for multi-process daemons. You come up with a whole buffet of ugly hacks to make that work most of the time. Then you install a program which doesn't come with a logrotate config. Then you decide you'd really want a more structured way to analyze logs than writing fragile regexes. Then you want to run something as a non-root user. Then you have an ephermeral batch job. Then you want to ship off all of your logs to another location and really don't want to duplicate the log paths yet again. Then you want to see what happened just before the server crashed last boot. Then you run two of a daemon and give them different log paths. Then you want to look at your kernel and application logs together. Then you want to delete just last month's logs to free up some space. Also wouldn't it be handy if there was some kind of index to make things faster? ...
- Arch-TK 6y ago
  All of this seems solved by svlogd honestly.
  - ATsch 6y ago
    I have never used svlogd but at it looks like it does read logs from it's stdin, which is a solid step forward as it avoids much of the pitfalls caused by applications writing directly into log files. Collection is just the first part of a log system though of course.
- magicalhippo 6y ago
  > However, removing bytes from the start of an open file is a bad idea. A digression, but I keep wondering why filesystems never optimized this scenario. Being able to trim the start of the file is really useful in certain cases, and without support options are pretty bad. It also seems fairly easy to support at the cost of a few bytes in the file entry (a start-of-data index into first sector), and resizing a file by seeking past end is already a thing. I must be missing something...
ex_amazon_sde 6y ago
JournalD is excellent. Indexing structured data like logs is the right thing to do. Log messages have a lot of useful metadata to do filtering on. > Systemd could completely eliminate the need for stuff like docker by totally isolating a process. It does: nspawn is much leaner than docker and it's available on all Linux systems with SystemD.
- stock_toaster 6y ago
  > JournalD is excellent This literally made me laugh out loud. Over the years I have historically had several production system outages due to journald corrupting its own data store, being unable to log, and services hanging due to being unable to log (process stdout buffer full, journald not reading it, so process stuck trying to write output). It certainly has improved over the years, and it has been a long time since this issue has happened (on one of the systems I manage at least), but I have accrued some pretty deep seated feelings about it at this point due to historical pain.
skynet-9000 6y ago
cgroups predate systemd by quite a lot (even if systemd absorbed yet another unwitting victim)

porsager 6y ago

That's interesting. Do you have a link to more info about isolating a process like that?

comboy 6y ago
Arch wiki is usually a good place to learn something in a succinct way (I've never even used Arch): https://wiki.archlinux.org/title/cgroups https://wiki.archlinux.org/title/cgroups

moshmosh 6y ago

The nice thing about docker is that you can use the same tooling on lots of operating systems, not just that it wraps some Linux functionality. Sure, it may involve using a VM, depending on the system, but I still get to just run "docker" and "docker-compose" and such, as if it were native. A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. Really, though, if I care about isolation and don't want to use docker, I'm gonna reach for FreeBSD + jails anyway, and I don't think systemd's gonna change that for me any time soon. If I want containers with better tooling outside the target OS, I'll use docker. Systemd falls in an awkward middle that I don't have much interest in.

vially 6y ago
> A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. But that's exactly what happens with docker anyway (e.g.: Docker is still running in the Linux VM when running it from a different operating system). And considering the widespread systemd adoption throughout the Linux ecosystem, it's far more likely for systemd to be already installed in the VM than docker anyway.
- moshmosh 6y ago
  Docker on Linux runs in a VM? No. Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though. Systemd-docker-replacement: works only on systemd-Linux. Tooling for any other arrangement, including non-systemd linux, left as an exercise to the user. Docker: works on Linux—systemd or otherwise. Has tooling to making working on macOS or Windows non-terrible and lets you use (mostly) the same commands as you would on Linux. I'm saying that cross-platform tooling is what makes Docker Docker. Systemd can't replace it without replacing the tooling, even if it can do the same thing only on systemd-Linux.
  - vially 5y ago
    > Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though. Would it? There's no reason this hypothetical systemd-docker-replacement couldn't be architected in a way that would not have a runtime dependency on systemd-the-init-system. In any case, I'm not even sure if systemd wants to be a docker replacement (although it does seem to pick up more and more container features lately). But there's definitely some overlap between the two projects (in particular around process/service management) but podman is a much more direct competitor to docker than systemd.

comboy 6y ago

It's just status quo. How would you feel about introducing having to log things into file after all you needed was stdout? Btw, you still can do that nothing is stopping you, but I don't think each and every service needs to decide (and let user configure) where to put its logs, how to separate them (maybe by errors, maybe by users/domains) and how to rotate them. I'm still not a big fan of systemd but it does make some things easier (but why wouldn't they think for a moment how easy "systemctl restoptart foo" will be to type, tab complete and issue different commands one after another)

dale_glass 6y ago

The logging is amazing. Transparent compression, easy search, output in JSON if you want to parse logs, timestamps with microseconds if you need them. No, writing stuff to a file was not fine. It's only fine if all you do is to 'tail' and 'grep' it, but any time you want to do something more advanced, it's a bloody mess. Why parse a text date, when it started its life as an unix timestamp? Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log that you could just search for? Why deal with parsing while the log may rotate or truncate under you, when the log system could be making your life easier? I wrote a whole bunch of log parsing stuff in my day, and so I have no nostalgia whatsoever regarding this "just write stuff to a file" approach.

Arch-TK 6y ago
> The logging is amazing. Amazing for you maybe. If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration). Meanwhile cat or tail -f of a log file is almost instant somehow. > Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log I mean, this is a problem which doesn't require systemd to solve, you just need a better log format. Someone could have standardised a line based logging format with fields and then someone else could have written libraries to write it and libraries to parse it. > Why deal with parsing while the log may rotate or truncate under you That's a log daemon problem that I haven't found to exist with any log daemon I have used.
- dale_glass 6y ago
  > If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration). That's because you're asking for a log of a service since logging began, which may well have been months worth of logging. You're saying "give me everything that happened ever to this service in a pager, then scroll down to the bottom". If you want tail -f, use a -f: 'journalctl -f -u <service>'. That's going to be a lot faster. You can also use '-b' to make it return a log of what happened since boot (worth looking into the manual for the details of that one), or specify a start date with -S, like '-S yesterday'. > Someone could have standardised a line based logging format And what were they waiting for? In the end, systemd did it, but it's a pretty recent invention. There was plenty time to get there first. > That's a log daemon problem that I haven't found to exist with any log daemon I have used. Because log daemons classically don't care about this at all -- it's a problem for whatever reads the files that the log daemon generates afterwards.

ATsch 6y ago

> Just writing stuff to a file was fine. Haha, I wish. Then that file becomes too large. So you want to truncate the start. However, removing bytes from the start of an open file is a bad idea. So you open a new log file every day instead. You really want to compress the old log files too, to save space. So now you can no longer just grep the logs as before. You're now also duplicating code and configuration for every daemon. So you write a daemon called `logrotate` that does this centrally instead. Into which you then copy the log paths configured for every program on your system and how long the logs should get kept. However programs don't much like it when you move the log files out from under them. So you patch every program respond to a signal by checking whether it's log files have been moved. Which then breaks for multi-process daemons. You come up with a whole buffet of ugly hacks to make that work most of the time. Then you install a program which doesn't come with a logrotate config. Then you decide you'd really want a more structured way to analyze logs than writing fragile regexes. Then you want to run something as a non-root user. Then you have an ephermeral batch job. Then you want to ship off all of your logs to another location and really don't want to duplicate the log paths yet again. Then you want to see what happened just before the server crashed last boot. Then you run two of a daemon and give them different log paths. Then you want to look at your kernel and application logs together. Then you want to delete just last month's logs to free up some space. Also wouldn't it be handy if there was some kind of index to make things faster? ...

Arch-TK 6y ago
All of this seems solved by svlogd honestly.
- ATsch 6y ago
  I have never used svlogd but at it looks like it does read logs from it's stdin, which is a solid step forward as it avoids much of the pitfalls caused by applications writing directly into log files. Collection is just the first part of a log system though of course.
magicalhippo 6y ago
> However, removing bytes from the start of an open file is a bad idea. A digression, but I keep wondering why filesystems never optimized this scenario. Being able to trim the start of the file is really useful in certain cases, and without support options are pretty bad. It also seems fairly easy to support at the cost of a few bytes in the file entry (a start-of-data index into first sector), and resizing a file by seeking past end is already a thing. I must be missing something...

ex_amazon_sde 6y ago

JournalD is excellent. Indexing structured data like logs is the right thing to do. Log messages have a lot of useful metadata to do filtering on. > Systemd could completely eliminate the need for stuff like docker by totally isolating a process. It does: nspawn is much leaner than docker and it's available on all Linux systems with SystemD.

stock_toaster 6y ago
> JournalD is excellent This literally made me laugh out loud. Over the years I have historically had several production system outages due to journald corrupting its own data store, being unable to log, and services hanging due to being unable to log (process stdout buffer full, journald not reading it, so process stuck trying to write output). It certainly has improved over the years, and it has been a long time since this issue has happened (on one of the systems I manage at least), but I have accrued some pretty deep seated feelings about it at this point due to historical pain.

skynet-9000 6y ago

cgroups predate systemd by quite a lot (even if systemd absorbed yet another unwitting victim)