9 ms·
A timesyncd failure and systemd's lack of debugability
- insaneirish 8y agoGiven things like this: "The default configuration is defined during compilation" [1] and the fact that you'd think 'timedatectl status' would tell you the NTP server you're supposedly trying to sync to (spoiler alert: it doesn't), I'm unsurprised of any further atrocities committed by systemd. [1]: https://www.freedesktop.org/software/systemd/man/timesyncd.conf.html https://www.freedesktop.org/software/systemd/man/timesyncd.c...
- tomjakubowski 8y ago> Given things like this: "The default configuration is defined during compilation [so a configuration file is only needed when it is necessary to deviate from those defaults....]" I don't get it. What's so bad about that?
- hedora 8y agoIt means that a configuration file on two similar machines could have radically different behaviors.
- jhasse 8y agoThat's always the case due to different software versions.
- TooBrokeToBeg 8y agoThis implies they can be different with the same versions.
- LoSboccacc 8y agoOnce you could just maintain some file from etc in a version control and setup a machine predictably with a script three lines long Then it came puppet chef and a shitload other stuff because ‘devops’ which make the task easier once and then fuck up your environment at every single upgrade changing defaults, changing dependencies and changing the environment you run I swear what could be done repeatedly the same way for a decade now it’s a frail mess that requires yearly fiddling to keep it running and current No wonder companies need so much devops people/time these days, they need them to keep following the frail shit that’s dropped into stable distro. Systemd is just one more thing that you have to reconfigure at every release because changes defaults subtly.
- JdeBP 8y agoYou might want to cast your mind back to https://news.ycombinator.com/item?id=9809427 https://news.ycombinator.com/item?id=9809427 .
- insaneirish 8y agoThe practical result is that on a machine without a configuration file, you can’t determine what NTP server it’s trying to use without having access to the source code (or some compile time include) or doing some other trickery. That is a ludicrous design decision and is indicative of the developers having zero empathy towards their users.
- agf 8y agoThe follow up article (https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynamicUserDangerous https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...) explains the underlying problem. Basically, to get around root-only readable directories, a special mount namespace is created when Dynamic Users are used. That fails when there are mount points that systemd can't access, which was due to FUSE filesystems mounted inside owner-only home directories, combined with an NFS mount. This is the second time in a few days that I've heard about systemd-related user / ownership problems. The other was related to when a systemd-managed socket needs to be owned by a user that is created by a daemon that relies on networking to already be up (ldap / activedirectory / whatever). systemd wants to create the socket at the same time as networking is starting, so the user doesn't exist yet -- and there isn't a simple way to tell systemd to delay creating that socket. Instead, the post-start script of the user daemon can chown the socket, and services relying on it will need to be started after that -- and there is no generic way to say to start a service after the auth provider is up, so you have to be specific, making the dependency fragile.
- LukeShu 8y ago> and there isn't a simple way to tell systemd to delay creating that socket. … and there is no generic way to say to start a service after the auth provider is up That's what After=nss-user-lookup.target is for; just stick that in the [Unit] section of the .socket file. See the systemd.special(7) man page for more information. ---- Edit: However, there is an interesting problem related to that: when you use socket activation for the user-providing daemon itself, you'll get a boot deadlock. For instance, one of the common LDAP solutions, nss-pam-ldapd, uses NSS and PAM modules that use a local socket to talk to a daemon (nslcd) that handles actually speaking LDAP. If you adjust that daemon & its service file to use socket activation (a trivial patch), the system will deadlock! Systemd will start the nslcd socket, but then when systemd starts dbus.service as a special hack it will then call dbus_init() to register itself on the bus. This will cause dbus-daemon to use the NSS module and hit the nslcd socket. Since this happens very early, systemd probably hasn't started nslcd itself yet. Because systemd is single threaded, and it's waiting on dbus-daemon to reply, it won't be able to handle the request to start nslcd, and the boot process deadlocks. Eventually, something will time out, and it will hobble along in to a half-working system. You can hack around this by manually ordering nslcd.service before dbus.service.
- modells 8y agoIt might be config and security fragile, but ntpd is a safe bet. When religions preferences come up, I prefer s6 because it’s scalable like daemon tools/runit. Old-school upstart was meh and ancient style rc scripts rarely did process supervision. Debugging and simplicity are prime requirements for init’s. http://www.skarnet.org/software/s6/ http://www.skarnet.org/software/s6/ PS: Logs aren’t files... they’re streams of structured messages often, that are unfortunately usually destructured into text lines. Log-rotation, logging to local files are horrible kludges that doesn’t scale for high-volume servers... log-structured log device, log sourcing or time-series structured db are more natural fits. Keeping log data in structured form instead of broken up in arbitrary line formats makes custom parsing unnecessary and log mining/transformation (ETL) much easier and usable. “Grepping” is simply looking at a text view of desired fields like a full-text db search. IOW, historical logs become random-access msgpack/protobufs streams while app logs are just always appending... and one string per log message for backwards compat, with a “logger”-type command for logfile-less 12-factor apps.
- erik_seaberg 8y agosystemd's job is to start daemons. It can be reasonable to decide dynamically when to start them based on which others are healthy (maybe you care more about speed than determinism on a laptop). But there's no reason to choose a uid and mkdir dynamically. You want these errors at install time, not at the box's next (probably unattended) restart.
- LukeShu 8y agoEverything that would have been "a weird problem with Foo Daemon's init script" we now call "a weird problem with systemd". So you think that systemd-timesyncd's service file shouldn't choose the uid at service-start time. That's not really the issue though. The issue is that for service isolation reasons (related to choosing the uid at service-start time), the service file wants to run systemd-timesyncd in a private mount namespace, but it is failing to properly set up that namespace. That's a more general issue, as there are a number of things that could trigger running the service in a private mount namespace; not just DynamicUser.
- erik_seaberg 8y agoIf SELinux doesn't cut it and you actually need to jail a badly-behaved service, "set up a jail in this specific way and exec this process in it" should be a tool to be invoked by an init script or a unit file or whatever. unshare(1) looks like it's halfway there.
- LukeShu 8y agoThat's exactly what's going on here. The systemd service launcher is doing the same thing as unshare(1); calling unshare(2). However, by "default" after you `unshare --mount`/`unshare(CLONE_NEWNS)`, changes made to existing mounts can propagate to the parent namespace, so then you need to do something like `mount --make-rslave /` to complete the isolation. As I understand the bug, the problem is in systemd's version of the `mount --make-rslave /` step, which (for a reason unknown to me) tries to stat (as root) the mountpoint first, and if the mountpoint is on a remote FS that might fail, and systemd isn't correctly handling that.
- andrewstuart 8y agoBefore systemd, a difficult bug was a difficult bug. With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist. I like systemd, it's a pity some vocal people aren't enthusiastic about it. Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking"? I guess my question would be, is this really a general issue about systemd being difficult to debug, or some specific component subsystem of systemd. systemd is a big place.
- agwa 8y agoIt's disappointing to see the top comment here condoning ad hominem attacks. But if you do believe that the identity of a speaker is more important than the arguments they make, note that the author of this blog post has a history of praising systemd (e.g. https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdRight https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdRight) There is no anti-systemd bias motivating this post.
- andrewstuart 8y ago>>condoning ad hominem attacks I'm not condoning ad-hominem attacks. In fact I am pointing out that often it is systemd itself that it the victim of ad-hominem attacks, which is to say that often people attack systemd, not its behaviors.
- esaym 8y agoI'm always amazed at what systemd can do that I don't know about. I had no idea about timesyncd. How did anyone find out about it? Is there a super awesome book or video of systemd "hacks" that I can read or watch? I think I've read the official docs but I don't think I really learned anything, but that was a couple of years ago. Edit: Wow, just realized that my debian laptop is running timesyncd. I run the same debian version on all my servers too but I installed ntpd on them. I guess I need to check how their conflicting, if any...
- mmagin 8y agoas a start, try "man -k systemd"
- oconnor663 8y ago> this server drifted more than half a second out of synchronization before I found this issue entirely by coincidence Did `systemctl status` not report "degraded" for this failure? Maybe the problem is that there was no monitoring set up for failing services?
- KaiserPro 8y agoI'd suggest that the time drift was tested directly. 0.5 seconds for most things is a reasonable threshold I'm pretty sure, by the way the OP described the problem that both the service state _and_ the product of the service are being monitored
- newnewpdro 8y agoTL;DR: systemd has bugs. Mountain out of a mole hill. Lack of debuggability? It's not like this is some threaded golang program you'll lose your mind attempting to make sense of in strace. systemd is a single-threaded C program, if you consider this as not debuggable it's your failing not systemd's. There are valid complaints to be made about systemd, but from what I've seen the crux of the problem is generally the haste with which its features are conceived and developed. For the most part it's a relatively conservative codebase, in a simple and easy to understand language without much of a runtime interfering with relating the code to the underlying system calls performed. This project doesn't have very many core contributors. It's somewhat incredible how much has been achieved with so little. But as a result, it's not perfect.
- steevdave 8y agoThe lack of contributors has a lot to do with Lennart's (apologies if I misspelled his name) attitude early on in systemd's development. I have no idea if it has changed, as I gave up trying to get things fixed. The one issue I still have with systemd is the way it handles the kernel's command line. I work with Chromebooks a lot, using software that isn't ChromeOS. The cool thing is that they can use FIT images so you can load multile kernels and device tree bindings into one image. So I can use the same sdcard on the Acer tegra Chromebook, Samsung's Chromebook, as well as the ASUS Flip Chromebook. However, to boot from sdcard you have to enable developer mode. When you enable developer mode, it adds a flag to the kernel command line "debug" - systemd sees that and thinks you want systemd in debug mode (wat?) Unfortunately there is no way to override this. I've tried passing the log level as info, which sort of works, until journalctl starts, and IT parses the debug flag in the command line and ignores the flag that sets the log level to info. No amount of explaining this helped and I (and many others who have had various issues) gave up on trying to help make systemd better. The only workaround to this is to stop using 1 sdcard amongst all of my Chromebooks and use individual sdcards per machine that uses a u-boot built for each Chromebook that does not pass the debug flag. I like systemd and many of it's features are helpful, but when you run into issues, it's almost always an uphill battle to fix them, even when you provide a patch.
- 8y ago
- thaumaturgy 8y agoLet's carefully step around the whole distraction that is systemd. Instead: > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. This seems to be really common practice in modern software development, and it sucks. There's a rush to rewrite old software and to match some or most of that software's feature set, enough to get people to adopt it, but useful error messages and other diagnostics represent a lot of extra work and so nobody bothers. Not enough programmers are spending time as systems administrators, trying to keep plates spinning, and so they aren't learning to appreciate the value of software that helps you troubleshoot it when something goes wrong. It's turning into the long-running conflict between mechanics and automotive engineers, where mechanics are tired of things like, "To replace [simple part that needs to be replaced every 5 years]: begin unbolting the entire front of the car. The front of the car uses 23 similar, but different, types of fasteners; keep track of them." It needs to be a rule that if somebody's going to rewrite some software, they've got to match or improve the logging and error handling in it. Error handling needs to start getting the kind of attention and shaming for bad behavior that bottom-barrel security practices gets.
- gcb0 8y agoThat's not quite right in this case. Systemd and gnome3 were cases where the new 'version' did not need full feature parity. Both those projects were written on leisure time, with no pressure for deadline or feature. and still adopted from top down. i guess some projects forget the benevolent part of benevolent dictator for life. systemd arguments were just "boot faster" and "it's newer". Everyone knew it was going to be broken, buggy, lacking features. As those were never actual goals. In fact, the goals had features never intended for init. So we could say they were rushing for feature anti-parity. https://www.tecmint.com/systemd-replaces-init-in-linux/ https://www.tecmint.com/systemd-replaces-init-in-linux/ see the table at the end. All the features systemd have show "no" for init. And "verbose debug", something you would consider very essential on a piece of software responsible for making your machine something broken and unusable, just show "no" for systemd, and nobody ever cared, because: boot faster (or so they say) systemd is a mistake. But i guess now it is a permanent mistake.
- otabdeveloper2 8y agoCome on, it's not like shell scripts are debugable.
- rleigh 8y agoErr, they are very easy to debug. You can have them print out every line as they execute and follow along what was done. Prior to systemd, if there was ever anything critically broken, I could step through the entire startup from the initramfs and bring up the system line-by-line or script by script as I needed, and see exactly what was at fault. With a script-based startup, the entire system was understandable by anyone with some rudimentary knowledge. It wasn't complicated, and it worked well. It did have its shortcomings, but the accessibility to modification and the ability to effectively debug were not amongst them. In comparison, systemd's behaviour is a black box with the unit files as input. The actual internal state is opaque, and it's inaccessible even to experts.
- reacweb 8y agoThis blog is full of explanations to understand how systemd can cause issues (like https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynamicUserDangerous https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...). This kind of information is often omited from official documentation but is so valuable when investigating bugs.
- iforgotpassword 8y agoI'm repeating myself, but the best way to use systemd is as an init system only, and then staying away from those fancy features like dynamic users (wtf) or socket activation. Especially avoid all that bolted on crap like timesyncd, resolved, etc. Yes they integrate a bit better with systemd, but no you don't need this better integration, especially since these tools are usually less configurable than their bigger brothers and are often more buggy just because they are not that battle tested yet. > systemd will tell me to reproduce on the latest version, Ubuntu will ignore it as always Amen to that. Distro bug reporting/handling is a pretty sad story on its own, and so far dealing with Ubuntu has been the worst. The most insane instance was when I reported a bug in their ati x11 driver. It was about two weeks after a new non-LTS came out, I even tracked down the bug and found that it was already fixed upstream only 2 commits after the one they shipped. So they didn't need to do anything but apply that single patch. No response ever. 6 months later the next release just shipped newer drivers and the problem was gone, but hadn't I recompiled the fixed driver myself I wouldn't have been able to switch vt for half a year without crashing X. I really like open source and all and make an effort to produce usable bug reports to help improve things for everyone, but instances like this make me wonder what the f*ck is wrong with people maintaining such projects. <end unrelated rant>
- majewsky 8y ago> Especially avoid all that bolted on crap like timesyncd, resolved, etc. You're pretty much stuck with systemd-resolved when you use systemd-networkd. And systemd-networkd is a godsend. It's the first network setup tool that I've found that is: - full-featured - simple to set up for the 99% usecases like "just do DHCP whenever a physical interfaces goes UP" - available and works the same on all major distros - not a Lovecraftian monstrosity like network-manager
- fps_doug 8y agoAgree, this is the only thing that's almost as well-designed as the init part of systemd. Unfortunately, you still have some weird edge-case every now and then with it that's just not possible to set up, and then you're screwed because you cannot just plug in in a little shell snippet somewhere.
- deleted 8y ago[deleted]
- majewsky 8y ago> this server drifted more than half a second out of synchronization before I found this issue entirely by coincidence Sounds like time synchronization is not particularly important for you. We have alerts set up for high clock drift that trigger at abs(drift) >= 300ms.
- mrweasel 8y agoThe more I use Systemd the more I actually like it. It has some weirdness to it and there are parts that aren't really useful. Journald is pretty stupid when you think about it. Either you're setup is really small, and logging to text files by defaults would be simpler to use. Alternatively your setup is large enough that you have a log-host/log-analytics, and logging to text files would be a sane default, as you need to convert the binary log to text anyway to do log-shipping. It hard to argue that writing startup script hasn't improved. I would much rather write a service file than an up-start script.
- beagle3 8y agoI have switched between ntpd, openntpd, timesyncd and a couple of others I might have forgotten, and the only one that's worked reliably is ntpd, but I was trying to stay off it, because historically it had quite a few vulnerabilities. I manage hundred of stand-along edge devices, mostly hardened PCs who have to bear up to 50 deg celsius environment from time to time; I've observered 5-10 times that RTCs were wrong by 5-10 hours in some direction after a power-loss, and openntpd fails miserably, timesyncd also not-so-good. ntp does ok. It's been much better since I started updating the rtc every 10 minutes from the system clock - most distros update from rtc on startup and to rtc on shutdown, but if the shutdown is after 6 months due to power loss, you're left with an rtc drift, which -- in harsh environment -- can be significant. I am right now testing chrony, which seems to be a winner, but it's too early to tell.
- deleted 8y ago[deleted]