6 ms·

Systemd auto-restarts of units can hide problems from you

108 points by ingve 3y ago

KingOfCoders 3y ago
But it can also make things more stable. I laughed when 30 years ago people recommended to restart Windows NT to make it more stable. I now have some Go websites running for a long time with daily Systemd restarts.
- justinclift 3y ago
  > I now have some Go websites running for a long time with daily Systemd restarts. Are the restarts needed because there's malicious over-size content being sent to it, trying to exhaust the server resources? If so, then "http.MaxBytesReader" might be helpful: https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f59b25fb78a576c0cac429323cad/webui/main.go#L5447 https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...
  - ilyt 3y ago
    I'm guessing he forgot null check
- gbraad 3y ago
  Edit: posted this as an independent comment
- nonameiguess 3y ago
  I believe this is one of the claims of chaos engineering. If you randomly restart services, you have to build them in a way that they're resilient to random failures. If you do this to entire servers, VM, or containers, whatever your unit of OS userspace is, you can also kill an attacker's foothold if they manage to get one. Sort of how if humans were capable of respawning in a healthy base state but with their memories intact, you could stop the spread of a disease by just killing everyone.
  - hashhar 3y ago
    You'll find Altered Carbon very very interesting then. Similar concepts being explored there for human "resleeving". https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series) https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series)
- regularfry 3y ago
  Heroku notably does (did? It's been a while) daily restarts. I believe they started it to take away the support hassle of folks whose apps had memory leaks, but it's also a convenient mechanism to allow cycling the infrastructure without needing live migration.
- KronisLV 3y ago
  > I now have some Go websites running for a long time with daily Systemd restarts. I went a step further and all of my servers do staggered restarts at the start of every month (previously daily/weekly, but once I worked most stuff out, a month is enough now). No more update related issues catching me off guard when I suddenly NEED to do a restart in short order, no issues regarding manual processes or a service that doesn't start automatically after a restart/crash, or other things that you might not catch otherwise, load balancing etc. Even stuff like restarting the Swarm cluster leader/followers close to one another, yet expecting them to eventually communicate with one another properly and schedule all of the containers, pull them and launch them as needed, join them to the networks and actually communicate properly to them. Knowing that restarts are inevitable (sometimes after a crash), might as well make sure that they work properly. And when you discover a new failure mode and something indeed doesn't work, it's nice to test your monitoring and alerting, as well as learn how to deal with that particular failure.
- toast0 3y ago
  Or it can make things worse. I tried a load balancer once that had a reboot once a month. Not worth it when the underlying service hosts had failures more like once every three months. Round robin DNS was better than that.
viraptor 3y ago
Another interesting thing you may want to check for is coredumpctl. At $job, I'm installing a custom handler for those, extracting the stacktrace and submitting it to Rollbar with the rest of the app events. It only fired once so far, but it's something you want to know about. (Same should be trivial to do with Sentry or other error reporting services)
shoo 3y ago
auto-restarting after a failure is a reasonable first order workaround for transient environment issues (network or external dependency goes away for a bit) or perhaps rare per-request issues that aren't handled correctly. arguably it'd be better to understand and fix or mitigate the issue properly, but until you actually have time to do that, it's nice for the service to autonomously attempt to be available. ...except when that causes even more problems! one example i remember from a hobby project was a worker service that would make requests to an external service's API. I'd implemented a throttle in the worker to limit the number of external requests per second made against the external service, where the throttle state was stored in process memory and not persisted anywhere -- seemed to be a pragmatic design tradeoff. you probably see where this is going. there was an exciting interaction where an external API response caused the worker's API client to throw an unhandled exception, which took down the worker service. then systemd would diligently restart the worker service immediately per the restart policy. the throttle working state had been lost, so upon being restarted the worker service immediately fired another request to the external API, which then issued the same API response, which caused the worker service to fail in the same way, ... luckily noticed by systemd and immediately restarted. combine with a lack of monitoring + alerting and you get a mechanism where your worker service can make about 100,000x as many external API requests over a few days as it was meant to.
- regecks 3y ago
  I always set `RestartSec`.
  - vidarh 3y ago
    That, plus restricting the number of restarts within an interval, is good. You can then also set "OnFailure" to trigger another unit if the failure state is reached, e.g. to trigger a notification. E.g.: [Unit] ... OnFailure=notify-failure@%n.service [Service] Type=simple Restart=on-failure RestartSec=5 .. StartLimitBurst=5 StartLimitIntervalSec=300
- ilyt 3y ago
  I mean, if you fail to code it correctly in the first place then fail to set up reasonable auto restart policy then fail to monitor it as well, yeah, shit will eventually break
gbraad 3y ago
I think they rather received a phonecall/pager on the weekend than having it restart. ;-) This is actually part of the self-healing aspect. One way to see if a service restarts is using the following command: sudo systemctl show [servicename].service -p NRestarts I prefer to have this set as Restart=on-failure Also note, there are OnFailure to trigger an additional log message, or recovery service, and FailureAction to allow actions to happen, such as the suggested 'reboot' ;-) For reference: https://www.freedesktop.org/software/systemd/man/systemd.unit.html https://www.freedesktop.org/software/systemd/man/systemd.uni...
- ilyt 3y ago
  sudo systemctl show [servicename].service -p NRestarts any way to do that on all services aside from looping ?
  - kelnos 3y ago
    I think a better way (as mentioned by the parent) is to use FailureAction to do something (like send an event to an alerting system). Granted, you have to modify all the .service files that you care about, which is annoying.
  - gbraad 3y ago
    You would be selective. For OS services you should be able to rely on them... You also don't want to deal with all of them. For this you could otherwise just use journalctl. You also have OOM issues, and this has its own facility.
evil-olive 3y ago
> Whether you want to monitor for this sort of thing (and how) is an open question. It's certainly possible that this is one of the times where your monitoring isn't going to be comprehensive, because it's infrequent enough, low impact enough, and hard enough to craft a specific alert. my preferred way to alert on this is to use the process start time that's included by default in most process-level Prometheus metrics (and is trivial to implement yourself, if you need to): > curl -s localhost:9100/metrics | grep process_start_time_seconds # HELP process_start_time_seconds Start time of the process since unix epoch in seconds. # TYPE process_start_time_seconds gauge process_start_time_seconds 1.68957669109e+09 on a stable system, this metric will be very close to static. you can feed it through the PromQL changes() function to get a count of how many restarts have happened in a given time window. in my experience, for anything with an "alert if it's down for X minutes" rule, you probably also want an "alert if it restarts N times in Y minutes" rule.
- captn3m0 3y ago
  A relevant alternative is the systemd_exporter, which will export metrics for various systemd services, including restart count: https://github.com/prometheus-community/systemd_exporter#metrics https://github.com/prometheus-community/systemd_exporter#met...
  - tetha 3y ago
    Mh, good points. I'll have to take a look to include a restart count in our zabbix systemd template. Should be a cheap metric to track with deduplication but valuable to create tickets upon if it changes too much in a short timeframe.
- vidarh 3y ago
  A whole lot of alerting is made better by generally considering whether to alert on large changes in in almost any kind of event, and sometimes even by measuring the changes in the rate of change, because it has a better chance of surviving "alert fatigue". "That process that restarts spuriously once in a while has restarted again" stops being noteworthy very quickly if it's deemed low priority, and often even might lead to alerting thresholds being change because it becomes a nuisance. "That process that restarted every now and again now restarts 5 time as often as it used to" on the other hand is a lot more likely to get attention.
pikahumu 3y ago
The reasonable way to notice is to have alerts for any unexpected restarts. Relying on noticing intermittent service disruption is bound to fail. And so is "remembering to check for this": > in the future I'm going to want to remember to check for this Whenever you think that sentence, you should notice this as a red flag and re-think your approach. You will forget. And if not you, then somebody else in your team. You need automation for things you can forget, otherwise your mental checklists will grow too large to handle and are just a distraction.
- thayne 3y ago
  What is the best way to set up those alerts? Specifically how do you set something up that knows if unexpected restarts happened? Is there a dbus or similar event you can listen for?
  - pnutjam 3y ago
    https://www.baeldung.com/linux/systemd-service-fail-notification https://www.baeldung.com/linux/systemd-service-fail-notifica...
  - cramjabsyn 3y ago
    Worth looking into prometheus. in basic form that'd be gathering metrics from https://github.com/prometheus-community/systemd_exporter https://github.com/prometheus-community/systemd_exporter on your hosts, and configure alerting in grafana or prometheus alertmanager to notify when a threshold is exceeded
- harry8 3y ago
  This is good advice imho. Away from things that can be automated and are important you have a checklist and tick things off with a pen. Add $this to the list. Atul Gawande is worth reading in general and on this topic. [1] He turned it into a book I haven't yet read. [1] https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.newyorker.com/magazine/2007/12/10/the-checklist
- bayindirh 3y ago
crabbone 3y ago
There are plenty of similar examples out there. Take, for instance, Kubernetes indefinitely failing to start a pod and just retrying while taking no action to notify whoever started the process (just hanging). So, I can see two different aspects to this problem: * Lack of alerts. * Lack of RCA automation. For the first one -- it would've been nice if systemd had a "brother" program that could be used for alerts, so that no custom solution was necessary and that services could properly report intermittent problems etc. The second is contingent on several factors: re-envisioning error handling, declarative debugging and general popularization of the concept. There are several major problems with error handling in system programming languages today. Due to poor language runtime design, programmers learn to think about any and every error as essentially fatal. Recovery is typically seen as impossible, and if there's any sort of recovery code put in place, it's usually the one that tries to do the cleanup and start fresh. It's never really about fixing the problem. So, popularizing something like Common Lisp restart system with the ability to traverse the program stack back to the failing frame would've been a good first step in this direction. Declarative debugging, on the other hand, could be taking another step forward, where it could be made into a separate program which describes a complex recovery scheme. I.e. the idea of declarative debugging is that the programmer needs to describe the program in terms of constraints that should be checked when the program fails to identify the problematic place. The step forward would be to add automation for the cases when the failed constraint is discovered.
- vbezhenar 3y ago
  Nobody would test their complex recovery schemes. Fail fast is the way.
  - marcosdumay 3y ago
    Yep, recovering is a losing proposition. Database management systems don't bother with it anymore, your almost stateless deamon has no hope of gaining anything from it. But wide logging and error identification are things too; and very useful ones. And those plug on the same spots that were created for error recovering. Because of this, those are also not very common on our current system software.
WesolyKubeczek 3y ago
I think the feature exists primarily because there is quite a lot of flaky software out there that gets „miraculously cured” with a restart. Not having to nanny them until a proper fix is a boon, especially when you know the proper fix isn’t coming ever, or is not possible. That said, there exist options to determine that restarts are happening too quickly, indicative of a misconfiguration or that it just won’t work, period. See Timeout*. If this is not what you want, you can always change the restart policy, or specify the maximum number of restarts. Could be, though, that some software ships with crappy defaults which make it crash all the time and systemd just always restarting it and reporting everything is ok. Not a problem of systemd per se, though.
j16sdiz 3y ago
I hate systemd, but this is something actually good. If you need alerts, you should config monitoring for the logs.
- masklinn 3y ago
  You can actually do that in systemd pretty easily: add an `OnFailure` handler to the unit file.
  - gbraad 3y ago
    Or use the more dramatic `FailureAction=reboot` ;-)
buster 3y ago
If you didn't even notice for a long time, it might be better to auto-restart then having a downtime on the weekend if some service crashes, right?
vbezhenar 3y ago
Lack of monitoring can hide the problems from you.
- gbraad 3y ago
  Good point; it is not that systemd (or restarts) hides it, but it is monitoring and log analysis that is lacking.
mprootdev 3y ago
I had the exact same challenge with my daemons/scripts that could crash on bugs of mine. My solution: (Shameless plug) Create a small daemon that monitors the systemd-journal and email me a summary every 5-minutes (if any and not on blacklist). If you're interested (it's BSD2-licensed): https://github.com/mpdroog/hfast/tree/master/contrib/deltajournal https://github.com/mpdroog/hfast/tree/master/contrib/deltajo...
hnbear 3y ago
This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. This is just another reason to track and alert on error rates. It also depends how sensitive you are to failures, and required availability. But in general this sounds more like: “you need observability on error rate”, more than anything else, and that includes notification and alerting on set thresholds that seem aberrant.
- yxre 3y ago
  Observability is so key for successful infrastructure. A poor infrastructure with good observability is going to be more successful with than good infrastructure and no observability
- throwaway073123 3y ago
  Auto restarting is actually intended to hide problems! ...from the user. The user, who probably isn't you if you're developing or administrating a service. Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it. The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).
- outworlder 3y ago
  > This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia. If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted? At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required. Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically. > and alerting on set thresholds that seem aberrant. I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.
siscia 3y ago
This is the exactly the reason why I use auto-restarts. I know there are problems in the software I am running, but I don't what to think about it. I want it hidden from myself.
- fourstepper 3y ago
  Exactly - as long as the delivered data is of "expected" quality, I don't (want to) care.
SkipperCat 3y ago
And tbis is why we have logging system. A decent automated query of your syslog output should be able to show you any flapping services. A setup like this gives you not only self-healing (as described in the post) but also visibility into erroneous systemd units. I always thought a good monitoring system had three components. Polling, streaming data and log analysis. A lot of times folks don't bother with the logging and miss issues like what is described in the post.
- spmurrayzzz 3y ago
  Yea I was a little surprised to see the comment "It wouldn't hurt to look at the logs or the metrics, just in case" paired with the previous comment "there's not quite enough information exposed in the Prometheus host agent's systemd metrics to make it easy". Not everything needs to be monitored in a single way via a single platform. A cron job that greps/awks syslog for relevant strings and sends a slack message, email, sms, something would be a step function in the right direction.
  - yabones 3y ago
    It's even easier than that, the functionality exists inside rsyslog. All you have to do is set up a template in /etc/rsyslog.d and point it towards a mail relay... if $syslogseverity <= 3 then { action(type="ommail" server="127.0.0.1" port="25" mailfrom="rsyslog@localhost" mailto="root@localhost" subject.template="mailSubject" template="mailBody" action.execonlyonceeveryinterval="3600") } While I generally like to dump all logs into elasticsearch/opensearch, it's also really handy to have this set up for one-off machines so that critical/error logs that usually come from things breaking & restarting still get seen. https://docbot.onetwoseven.one/services/syslog/ https://docbot.onetwoseven.one/services/syslog/
    - ilyt 3y ago
      Rsyslog also supports sending to elasticsearch IIRC.
  - 3y ago
andrewstuart 3y ago
Yeah this doesn’t even need the word “systemd” in it.
- kaba0 3y ago
  How else would it get upvoted to the frontpage, if not through the secret flamewar word?
  - NoZebra120vClip 3y ago
    Because it's written by an experienced and highly-respected academic systems administrator who's been continuously publishing his high-quality, ad-free blog for longer than Hacker News has been around.
- marcosdumay 3y ago
  Most people didn't do it before systemd made it the default. I imagine that's what triggered the problem for the author, and thus lead to the article. It was so common to not restart that many services now have a "clear error logs on startup" policy.
  - timetraveller26 3y ago
    Though systemd doesn't do it by default, you have to specify it on the service.
  - freedomben 3y ago
    Interesting, pre-systemd the best practice everywhere I worked was to always restart. IME that's kind of the point of a supervisor in the first place.
    - marcosdumay 3y ago
      You had to go out of your way to put your software under a supervisor. Most things were just launched by an init script and forgotten about.
kristianpaul 3y ago
Thats right, just keep an eye on error rates either from parsing logs to metrics or sending exit codes somewhere
brundolf 3y ago
> You do get the node_systemd_service_restart_total metric, which counts how many times a Restart= is triggered, but that doesn't necessarily say why Wouldn't journald have details? Of course those might not be hooked up to the main reporting flow, but the debugging info is available once you discover the problem
exabrial 3y ago
Not if you're monitoring your systems. True engineering means you have a feedback loop into your process. Otherwise you're 'just a programmer'.... :) You should have Telegraph running on every system you have collecting stats and sending it to InfluxDb with sample rate around 15s and +- 5s of jitter. Each critical process should have an uptime counter. You should create both Deadman alerts (No contact for 5m) and Uptime alerts (uptime falls below 60s). s/Telgraf|InfluxDb/YourMonitoringSystemOfChoice/ Bonus: If you're running JVMs, you can expose _all_ of the JVMs stats to Telegraf via Jolokia, which can run as a Java Agent and requires 0 code changes. The JVM has very very detailed stats about it's health available by default, making problems very visible (if you look for them).
- glenjamin 3y ago
  If you're only monitoring user-facing symptoms, and the application restarts quickly enough to not cause a blip in service, then: a) It's highly likely that your monitoring is not going to spot an occasionally restarting service b) It's probably a good choice not to waste cycles on that sort of monitoring
  - exabrial 3y ago
    Both of your assumptions are completely incorrect. One should _never_ "monitor" a system like that. Fortunately, the wise Unix wizzards of yesteryear already thought of the solution: uptime counters. An uptime counter counts the number of milliseconds a process is alive. So if it resets to a value lower than the previous recorded one, it's a blue star thats impossible to miss.
    - ilyt 3y ago
      Eh, just boot time is better one, doesn't change so it doesn't eat that many bytes in whatever TSDB you use
      - exabrial 3y ago
        TSDBs don't work like that. They generally will choose to store a slope, not absolute values. They're very efficient at storing tons of data like this.
felixgallo 3y ago
systemd sucks, but this feature isn't the worst thing in the world, even if it's primitive. Probably the best implementation I've seen is in erlang ( https://www.erlang.org/doc/design_principles/sup_princ.html https://www.erlang.org/doc/design_principles/sup_princ.html ) in which supervisor processes get notified of child death and can use several different strategies to figure out what to do next (e.g. die and expect the supervisor's supervisor to handle it, terminate all children and restart, ...) Although you do lose state when a process dies, if your processes are sufficiently lightweight and/or stateless, it's possible for the problem to be irrelevant until the next software release can garden/amend/curate it. The win that you get in erlang/elixir/etc. is that you can just code the happy path most of the time, and then figure out which sad paths are the ones you need to deal with once you've seen production under load. If you've ever programmed in go, you understand how freeing that is.
- freedomben 3y ago
  systemd has nothing to do with this. In erlang/elixir if you configure your processes to automatically restart and you never check your logs, you'll have exactly the same issue.
throw7 3y ago
Light weight journal trawling: https://pypi.org/project/journal-brief/ https://pypi.org/project/journal-brief/
wkat4242 3y ago
I feel it's the same with the whole kerberos / "cattle not pets" cloud mindset. Usually when a service conks out, operations will just kill it and roll out another instance. Nice, but you have no way of finding out what actually happened then. Sometimes this is caused by someone looking for exploits (and actually finding one), or other stuff you'd really rather know about.
- _ea1k 3y ago
  You mean kubernetes and not kerberos?
  - wkat4242 3y ago
    Oh yes kubernetes sorry!
- chronid 3y ago
  You need better observability or tooling then. Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.
  - ilyt 3y ago
    You achieve availability by redundancy, not by running around with shotgun murdering your cattle.
    - chronid 3y ago
      Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis. If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers. In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.
cramjabsyn 3y ago
Ideally systemd metrics are being gathered and notifications are set up in a way that bring attention to services that are being restarted excessively
- deleted 3y ago
  [deleted]
- Nmi2osv7 3y ago
  [flagged]
Nmi2osv7 3y ago
is there some kind of gaming going on where everything this guy posts gets to the front page? he has 11 front page low-quality posts in 14 days
- freedomben 3y ago
  Looking at the history, I don't think so (though he has high karma value, I don't know if that is considered by the algo since it's secret). His failure rate is pretty typical. He just submits a lot of posts.
  - Nmi2osv7 3y ago
    No I'm talking about the blog author
- JdeBP 3y ago
  https://news.ycombinator.com/item?id=36957558 https://news.ycombinator.com/item?id=36957558 answers your question.
  - Nmi2osv7 3y ago
    Nothing about the 4-paragraph blog posts he makes is advanced. It is in fact a remarkably basic lesson of "monitor your services" that even a 16 year old developer would know. I think he is just used to talking down to idiots because of his job, maybe. No offense to the dude if he happens to be reading this, but less quantity and more quality please. I'm sure you're very smart. Alternatively, maybe HN can let us blacklist domains?
timetraveller26 3y ago
This is really confusing when I am trying to debug why a service failed and checking the logs the last errors are because the service is re-starting too quickly.
marcrosoft 3y ago
This is why OpenBSD’s rc doesn’t have restarts. A crashing application has a bug that needs fixed. A crashing application probably has a security vuln if nothing else it has denial of service.
- SoftTalker 3y ago
  Yes this is my default --- let it crash, and figure out why, and fix the problem. It also helps identify services that nobody really uses. If a service is down, and nobody complains, maybe it's not needed at all. Sometimes there are external demands that require the system to always be up or to minimize any downtime. In an ideal world, that requirement should be designed into the system architecture from the beginning, but in the real world that doesn't always happen.
  - tinus_hn 3y ago
    > Sometimes there are external demands that require the system to always be up or to minimize any downtime Then the solution is redundancy. Especially with todays frequent downtime for security updates you can’t deliver a singular system that’s always up.
- kelnos 3y ago
  > A crashing application has a bug that needs fixed. Sure, but you can fix that bug after it's been auto-restarted. Assuming this is a service that, say, customers rely on, your customers don't care about the details. They just care that they're able to use your service and get their work done. Not having some sort of auto-restart or auto-healing system only serves to prolong downtime. Obviously you need to be alerted of these restarts so you can find the bug and fix it, but overall I'd value uptime over the fairly low probability that allowing it to auto-restart might expose some sort of security issue. Of course there may be some circumstances where reliability is by far the biggest concern, so allowing something to crash and stay down may be preferable. I just think that's going to be a small minority of situations, for most people.
- kazinator 3y ago
  > if nothing else it has denial of service. Well, obviously so if running under OpenBSD's rc system. Pretty much all fault-tolerant systems feature recovery by strategies like auto-restarting things and raising alarms, so the OpenBSD people must know some secret that has eluded researchers in that area.
crest 3y ago
It's a trade-off between availability and correctness. Do you want anything buggy to crash and stay down until the the problem corrected or a supervisor attempt to run the same buggy code again hoping the bug isn't encountered again? There is no one-size-fits-all answer, but just restarting without reporting and counting the error rate is never a good decision.
spullara 3y ago
I think that is the point.
StimDeck 3y ago
[dead]
andrewstuart 3y ago
I design my applications around systemds ability to restart applications quickly and cleanly.

108 points by ingve 3y ago

But it can also make things more stable. I laughed when 30 years ago people recommended to restart Windows NT to make it more stable. I now have some Go websites running for a long time with daily Systemd restarts.

justinclift 3y ago
> I now have some Go websites running for a long time with daily Systemd restarts. Are the restarts needed because there's malicious over-size content being sent to it, trying to exhaust the server resources? If so, then "http.MaxBytesReader" might be helpful: https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f59b25fb78a576c0cac429323cad/webui/main.go#L5447 https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...
- ilyt 3y ago
  I'm guessing he forgot null check
gbraad 3y ago
Edit: posted this as an independent comment
nonameiguess 3y ago
I believe this is one of the claims of chaos engineering. If you randomly restart services, you have to build them in a way that they're resilient to random failures. If you do this to entire servers, VM, or containers, whatever your unit of OS userspace is, you can also kill an attacker's foothold if they manage to get one. Sort of how if humans were capable of respawning in a healthy base state but with their memories intact, you could stop the spread of a disease by just killing everyone.
- hashhar 3y ago
  You'll find Altered Carbon very very interesting then. Similar concepts being explored there for human "resleeving". https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series) https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series)
regularfry 3y ago
Heroku notably does (did? It's been a while) daily restarts. I believe they started it to take away the support hassle of folks whose apps had memory leaks, but it's also a convenient mechanism to allow cycling the infrastructure without needing live migration.
KronisLV 3y ago
> I now have some Go websites running for a long time with daily Systemd restarts. I went a step further and all of my servers do staggered restarts at the start of every month (previously daily/weekly, but once I worked most stuff out, a month is enough now). No more update related issues catching me off guard when I suddenly NEED to do a restart in short order, no issues regarding manual processes or a service that doesn't start automatically after a restart/crash, or other things that you might not catch otherwise, load balancing etc. Even stuff like restarting the Swarm cluster leader/followers close to one another, yet expecting them to eventually communicate with one another properly and schedule all of the containers, pull them and launch them as needed, join them to the networks and actually communicate properly to them. Knowing that restarts are inevitable (sometimes after a crash), might as well make sure that they work properly. And when you discover a new failure mode and something indeed doesn't work, it's nice to test your monitoring and alerting, as well as learn how to deal with that particular failure.
toast0 3y ago
Or it can make things worse. I tried a load balancer once that had a reboot once a month. Not worth it when the underlying service hosts had failures more like once every three months. Round robin DNS was better than that.

viraptor 3y ago

Another interesting thing you may want to check for is coredumpctl. At $job, I'm installing a custom handler for those, extracting the stacktrace and submitting it to Rollbar with the rest of the app events. It only fired once so far, but it's something you want to know about. (Same should be trivial to do with Sentry or other error reporting services)

shoo 3y ago

auto-restarting after a failure is a reasonable first order workaround for transient environment issues (network or external dependency goes away for a bit) or perhaps rare per-request issues that aren't handled correctly. arguably it'd be better to understand and fix or mitigate the issue properly, but until you actually have time to do that, it's nice for the service to autonomously attempt to be available. ...except when that causes even more problems! one example i remember from a hobby project was a worker service that would make requests to an external service's API. I'd implemented a throttle in the worker to limit the number of external requests per second made against the external service, where the throttle state was stored in process memory and not persisted anywhere -- seemed to be a pragmatic design tradeoff. you probably see where this is going. there was an exciting interaction where an external API response caused the worker's API client to throw an unhandled exception, which took down the worker service. then systemd would diligently restart the worker service immediately per the restart policy. the throttle working state had been lost, so upon being restarted the worker service immediately fired another request to the external API, which then issued the same API response, which caused the worker service to fail in the same way, ... luckily noticed by systemd and immediately restarted. combine with a lack of monitoring + alerting and you get a mechanism where your worker service can make about 100,000x as many external API requests over a few days as it was meant to.

regecks 3y ago
I always set `RestartSec`.
- vidarh 3y ago
  That, plus restricting the number of restarts within an interval, is good. You can then also set "OnFailure" to trigger another unit if the failure state is reached, e.g. to trigger a notification. E.g.: [Unit] ... OnFailure=notify-failure@%n.service [Service] Type=simple Restart=on-failure RestartSec=5 .. StartLimitBurst=5 StartLimitIntervalSec=300
ilyt 3y ago
I mean, if you fail to code it correctly in the first place then fail to set up reasonable auto restart policy then fail to monitor it as well, yeah, shit will eventually break

gbraad 3y ago

I think they rather received a phonecall/pager on the weekend than having it restart. ;-) This is actually part of the self-healing aspect. One way to see if a service restarts is using the following command: sudo systemctl show [servicename].service -p NRestarts I prefer to have this set as Restart=on-failure Also note, there are OnFailure to trigger an additional log message, or recovery service, and FailureAction to allow actions to happen, such as the suggested 'reboot' ;-) For reference: https://www.freedesktop.org/software/systemd/man/systemd.unit.html https://www.freedesktop.org/software/systemd/man/systemd.uni...

ilyt 3y ago
sudo systemctl show [servicename].service -p NRestarts any way to do that on all services aside from looping ?
- kelnos 3y ago
  I think a better way (as mentioned by the parent) is to use FailureAction to do something (like send an event to an alerting system). Granted, you have to modify all the .service files that you care about, which is annoying.
- gbraad 3y ago
  You would be selective. For OS services you should be able to rely on them... You also don't want to deal with all of them. For this you could otherwise just use journalctl. You also have OOM issues, and this has its own facility.

evil-olive 3y ago

> Whether you want to monitor for this sort of thing (and how) is an open question. It's certainly possible that this is one of the times where your monitoring isn't going to be comprehensive, because it's infrequent enough, low impact enough, and hard enough to craft a specific alert. my preferred way to alert on this is to use the process start time that's included by default in most process-level Prometheus metrics (and is trivial to implement yourself, if you need to): > curl -s localhost:9100/metrics | grep process_start_time_seconds # HELP process_start_time_seconds Start time of the process since unix epoch in seconds. # TYPE process_start_time_seconds gauge process_start_time_seconds 1.68957669109e+09 on a stable system, this metric will be very close to static. you can feed it through the PromQL changes() function to get a count of how many restarts have happened in a given time window. in my experience, for anything with an "alert if it's down for X minutes" rule, you probably also want an "alert if it restarts N times in Y minutes" rule.

captn3m0 3y ago
A relevant alternative is the systemd_exporter, which will export metrics for various systemd services, including restart count: https://github.com/prometheus-community/systemd_exporter#metrics https://github.com/prometheus-community/systemd_exporter#met...
- tetha 3y ago
  Mh, good points. I'll have to take a look to include a restart count in our zabbix systemd template. Should be a cheap metric to track with deduplication but valuable to create tickets upon if it changes too much in a short timeframe.
vidarh 3y ago
A whole lot of alerting is made better by generally considering whether to alert on large changes in in almost any kind of event, and sometimes even by measuring the changes in the rate of change, because it has a better chance of surviving "alert fatigue". "That process that restarts spuriously once in a while has restarted again" stops being noteworthy very quickly if it's deemed low priority, and often even might lead to alerting thresholds being change because it becomes a nuisance. "That process that restarted every now and again now restarts 5 time as often as it used to" on the other hand is a lot more likely to get attention.

pikahumu 3y ago

The reasonable way to notice is to have alerts for any unexpected restarts. Relying on noticing intermittent service disruption is bound to fail. And so is "remembering to check for this": > in the future I'm going to want to remember to check for this Whenever you think that sentence, you should notice this as a red flag and re-think your approach. You will forget. And if not you, then somebody else in your team. You need automation for things you can forget, otherwise your mental checklists will grow too large to handle and are just a distraction.

thayne 3y ago
What is the best way to set up those alerts? Specifically how do you set something up that knows if unexpected restarts happened? Is there a dbus or similar event you can listen for?
- pnutjam 3y ago
  https://www.baeldung.com/linux/systemd-service-fail-notification https://www.baeldung.com/linux/systemd-service-fail-notifica...
- cramjabsyn 3y ago
  Worth looking into prometheus. in basic form that'd be gathering metrics from https://github.com/prometheus-community/systemd_exporter https://github.com/prometheus-community/systemd_exporter on your hosts, and configure alerting in grafana or prometheus alertmanager to notify when a threshold is exceeded
harry8 3y ago
This is good advice imho. Away from things that can be automated and are important you have a checklist and tick things off with a pen. Add $this to the list. Atul Gawande is worth reading in general and on this topic. [1] He turned it into a book I haven't yet read. [1] https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.newyorker.com/magazine/2007/12/10/the-checklist
bayindirh 3y ago

crabbone 3y ago

There are plenty of similar examples out there. Take, for instance, Kubernetes indefinitely failing to start a pod and just retrying while taking no action to notify whoever started the process (just hanging). So, I can see two different aspects to this problem: * Lack of alerts. * Lack of RCA automation. For the first one -- it would've been nice if systemd had a "brother" program that could be used for alerts, so that no custom solution was necessary and that services could properly report intermittent problems etc. The second is contingent on several factors: re-envisioning error handling, declarative debugging and general popularization of the concept. There are several major problems with error handling in system programming languages today. Due to poor language runtime design, programmers learn to think about any and every error as essentially fatal. Recovery is typically seen as impossible, and if there's any sort of recovery code put in place, it's usually the one that tries to do the cleanup and start fresh. It's never really about fixing the problem. So, popularizing something like Common Lisp restart system with the ability to traverse the program stack back to the failing frame would've been a good first step in this direction. Declarative debugging, on the other hand, could be taking another step forward, where it could be made into a separate program which describes a complex recovery scheme. I.e. the idea of declarative debugging is that the programmer needs to describe the program in terms of constraints that should be checked when the program fails to identify the problematic place. The step forward would be to add automation for the cases when the failed constraint is discovered.

vbezhenar 3y ago
Nobody would test their complex recovery schemes. Fail fast is the way.
- marcosdumay 3y ago
  Yep, recovering is a losing proposition. Database management systems don't bother with it anymore, your almost stateless deamon has no hope of gaining anything from it. But wide logging and error identification are things too; and very useful ones. And those plug on the same spots that were created for error recovering. Because of this, those are also not very common on our current system software.

WesolyKubeczek 3y ago

I think the feature exists primarily because there is quite a lot of flaky software out there that gets „miraculously cured” with a restart. Not having to nanny them until a proper fix is a boon, especially when you know the proper fix isn’t coming ever, or is not possible. That said, there exist options to determine that restarts are happening too quickly, indicative of a misconfiguration or that it just won’t work, period. See Timeout*. If this is not what you want, you can always change the restart policy, or specify the maximum number of restarts. Could be, though, that some software ships with crappy defaults which make it crash all the time and systemd just always restarting it and reporting everything is ok. Not a problem of systemd per se, though.

j16sdiz 3y ago

I hate systemd, but this is something actually good. If you need alerts, you should config monitoring for the logs.

masklinn 3y ago
You can actually do that in systemd pretty easily: add an `OnFailure` handler to the unit file.
- gbraad 3y ago
  Or use the more dramatic `FailureAction=reboot` ;-)

buster 3y ago

If you didn't even notice for a long time, it might be better to auto-restart then having a downtime on the weekend if some service crashes, right?

vbezhenar 3y ago

Lack of monitoring can hide the problems from you.

gbraad 3y ago
Good point; it is not that systemd (or restarts) hides it, but it is monitoring and log analysis that is lacking.

mprootdev 3y ago

I had the exact same challenge with my daemons/scripts that could crash on bugs of mine. My solution: (Shameless plug) Create a small daemon that monitors the systemd-journal and email me a summary every 5-minutes (if any and not on blacklist). If you're interested (it's BSD2-licensed): https://github.com/mpdroog/hfast/tree/master/contrib/deltajournal https://github.com/mpdroog/hfast/tree/master/contrib/deltajo...

hnbear 3y ago

This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. This is just another reason to track and alert on error rates. It also depends how sensitive you are to failures, and required availability. But in general this sounds more like: “you need observability on error rate”, more than anything else, and that includes notification and alerting on set thresholds that seem aberrant.

yxre 3y ago
Observability is so key for successful infrastructure. A poor infrastructure with good observability is going to be more successful with than good infrastructure and no observability
throwaway073123 3y ago
Auto restarting is actually intended to hide problems! ...from the user. The user, who probably isn't you if you're developing or administrating a service. Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it. The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).
outworlder 3y ago
> This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia. If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted? At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required. Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically. > and alerting on set thresholds that seem aberrant. I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.

siscia 3y ago

This is the exactly the reason why I use auto-restarts. I know there are problems in the software I am running, but I don't what to think about it. I want it hidden from myself.

fourstepper 3y ago
Exactly - as long as the delivered data is of "expected" quality, I don't (want to) care.

SkipperCat 3y ago

And tbis is why we have logging system. A decent automated query of your syslog output should be able to show you any flapping services. A setup like this gives you not only self-healing (as described in the post) but also visibility into erroneous systemd units. I always thought a good monitoring system had three components. Polling, streaming data and log analysis. A lot of times folks don't bother with the logging and miss issues like what is described in the post.

spmurrayzzz 3y ago
Yea I was a little surprised to see the comment "It wouldn't hurt to look at the logs or the metrics, just in case" paired with the previous comment "there's not quite enough information exposed in the Prometheus host agent's systemd metrics to make it easy". Not everything needs to be monitored in a single way via a single platform. A cron job that greps/awks syslog for relevant strings and sends a slack message, email, sms, something would be a step function in the right direction.
- yabones 3y ago
  It's even easier than that, the functionality exists inside rsyslog. All you have to do is set up a template in /etc/rsyslog.d and point it towards a mail relay... if $syslogseverity <= 3 then { action(type="ommail" server="127.0.0.1" port="25" mailfrom="rsyslog@localhost" mailto="root@localhost" subject.template="mailSubject" template="mailBody" action.execonlyonceeveryinterval="3600") } While I generally like to dump all logs into elasticsearch/opensearch, it's also really handy to have this set up for one-off machines so that critical/error logs that usually come from things breaking & restarting still get seen. https://docbot.onetwoseven.one/services/syslog/ https://docbot.onetwoseven.one/services/syslog/
  - ilyt 3y ago
    Rsyslog also supports sending to elasticsearch IIRC.
- 3y ago

andrewstuart 3y ago

Yeah this doesn’t even need the word “systemd” in it.

kaba0 3y ago
How else would it get upvoted to the frontpage, if not through the secret flamewar word?
- NoZebra120vClip 3y ago
  Because it's written by an experienced and highly-respected academic systems administrator who's been continuously publishing his high-quality, ad-free blog for longer than Hacker News has been around.
marcosdumay 3y ago
Most people didn't do it before systemd made it the default. I imagine that's what triggered the problem for the author, and thus lead to the article. It was so common to not restart that many services now have a "clear error logs on startup" policy.
- timetraveller26 3y ago
  Though systemd doesn't do it by default, you have to specify it on the service.
- freedomben 3y ago
  Interesting, pre-systemd the best practice everywhere I worked was to always restart. IME that's kind of the point of a supervisor in the first place.
  - marcosdumay 3y ago
    You had to go out of your way to put your software under a supervisor. Most things were just launched by an init script and forgotten about.

kristianpaul 3y ago

Thats right, just keep an eye on error rates either from parsing logs to metrics or sending exit codes somewhere

brundolf 3y ago

> You do get the node_systemd_service_restart_total metric, which counts how many times a Restart= is triggered, but that doesn't necessarily say why Wouldn't journald have details? Of course those might not be hooked up to the main reporting flow, but the debugging info is available once you discover the problem

exabrial 3y ago

Not if you're monitoring your systems. True engineering means you have a feedback loop into your process. Otherwise you're 'just a programmer'.... :) You should have Telegraph running on every system you have collecting stats and sending it to InfluxDb with sample rate around 15s and +- 5s of jitter. Each critical process should have an uptime counter. You should create both Deadman alerts (No contact for 5m) and Uptime alerts (uptime falls below 60s). s/Telgraf|InfluxDb/YourMonitoringSystemOfChoice/ Bonus: If you're running JVMs, you can expose _all_ of the JVMs stats to Telegraf via Jolokia, which can run as a Java Agent and requires 0 code changes. The JVM has very very detailed stats about it's health available by default, making problems very visible (if you look for them).

glenjamin 3y ago
If you're only monitoring user-facing symptoms, and the application restarts quickly enough to not cause a blip in service, then: a) It's highly likely that your monitoring is not going to spot an occasionally restarting service b) It's probably a good choice not to waste cycles on that sort of monitoring
- exabrial 3y ago
  Both of your assumptions are completely incorrect. One should _never_ "monitor" a system like that. Fortunately, the wise Unix wizzards of yesteryear already thought of the solution: uptime counters. An uptime counter counts the number of milliseconds a process is alive. So if it resets to a value lower than the previous recorded one, it's a blue star thats impossible to miss.
  - ilyt 3y ago
    Eh, just boot time is better one, doesn't change so it doesn't eat that many bytes in whatever TSDB you use
    - exabrial 3y ago
      TSDBs don't work like that. They generally will choose to store a slope, not absolute values. They're very efficient at storing tons of data like this.

felixgallo 3y ago

systemd sucks, but this feature isn't the worst thing in the world, even if it's primitive. Probably the best implementation I've seen is in erlang ( https://www.erlang.org/doc/design_principles/sup_princ.html https://www.erlang.org/doc/design_principles/sup_princ.html ) in which supervisor processes get notified of child death and can use several different strategies to figure out what to do next (e.g. die and expect the supervisor's supervisor to handle it, terminate all children and restart, ...) Although you do lose state when a process dies, if your processes are sufficiently lightweight and/or stateless, it's possible for the problem to be irrelevant until the next software release can garden/amend/curate it. The win that you get in erlang/elixir/etc. is that you can just code the happy path most of the time, and then figure out which sad paths are the ones you need to deal with once you've seen production under load. If you've ever programmed in go, you understand how freeing that is.

freedomben 3y ago
systemd has nothing to do with this. In erlang/elixir if you configure your processes to automatically restart and you never check your logs, you'll have exactly the same issue.

throw7 3y ago

Light weight journal trawling: https://pypi.org/project/journal-brief/ https://pypi.org/project/journal-brief/

wkat4242 3y ago

I feel it's the same with the whole kerberos / "cattle not pets" cloud mindset. Usually when a service conks out, operations will just kill it and roll out another instance. Nice, but you have no way of finding out what actually happened then. Sometimes this is caused by someone looking for exploits (and actually finding one), or other stuff you'd really rather know about.

_ea1k 3y ago
You mean kubernetes and not kerberos?
- wkat4242 3y ago
  Oh yes kubernetes sorry!
chronid 3y ago
You need better observability or tooling then. Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.
- ilyt 3y ago
  You achieve availability by redundancy, not by running around with shotgun murdering your cattle.
  - chronid 3y ago
    Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis. If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers. In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.

cramjabsyn 3y ago

Ideally systemd metrics are being gathered and notifications are set up in a way that bring attention to services that are being restarted excessively

deleted 3y ago
[deleted]
Nmi2osv7 3y ago
[flagged]

Nmi2osv7 3y ago

is there some kind of gaming going on where everything this guy posts gets to the front page? he has 11 front page low-quality posts in 14 days

freedomben 3y ago
Looking at the history, I don't think so (though he has high karma value, I don't know if that is considered by the algo since it's secret). His failure rate is pretty typical. He just submits a lot of posts.
- Nmi2osv7 3y ago
  No I'm talking about the blog author
JdeBP 3y ago
https://news.ycombinator.com/item?id=36957558 https://news.ycombinator.com/item?id=36957558 answers your question.
- Nmi2osv7 3y ago
  Nothing about the 4-paragraph blog posts he makes is advanced. It is in fact a remarkably basic lesson of "monitor your services" that even a 16 year old developer would know. I think he is just used to talking down to idiots because of his job, maybe. No offense to the dude if he happens to be reading this, but less quantity and more quality please. I'm sure you're very smart. Alternatively, maybe HN can let us blacklist domains?

timetraveller26 3y ago

This is really confusing when I am trying to debug why a service failed and checking the logs the last errors are because the service is re-starting too quickly.

marcrosoft 3y ago

This is why OpenBSD’s rc doesn’t have restarts. A crashing application has a bug that needs fixed. A crashing application probably has a security vuln if nothing else it has denial of service.

SoftTalker 3y ago
Yes this is my default --- let it crash, and figure out why, and fix the problem. It also helps identify services that nobody really uses. If a service is down, and nobody complains, maybe it's not needed at all. Sometimes there are external demands that require the system to always be up or to minimize any downtime. In an ideal world, that requirement should be designed into the system architecture from the beginning, but in the real world that doesn't always happen.
- tinus_hn 3y ago
  > Sometimes there are external demands that require the system to always be up or to minimize any downtime Then the solution is redundancy. Especially with todays frequent downtime for security updates you can’t deliver a singular system that’s always up.
kelnos 3y ago
> A crashing application has a bug that needs fixed. Sure, but you can fix that bug after it's been auto-restarted. Assuming this is a service that, say, customers rely on, your customers don't care about the details. They just care that they're able to use your service and get their work done. Not having some sort of auto-restart or auto-healing system only serves to prolong downtime. Obviously you need to be alerted of these restarts so you can find the bug and fix it, but overall I'd value uptime over the fairly low probability that allowing it to auto-restart might expose some sort of security issue. Of course there may be some circumstances where reliability is by far the biggest concern, so allowing something to crash and stay down may be preferable. I just think that's going to be a small minority of situations, for most people.
kazinator 3y ago
> if nothing else it has denial of service. Well, obviously so if running under OpenBSD's rc system. Pretty much all fault-tolerant systems feature recovery by strategies like auto-restarting things and raising alarms, so the OpenBSD people must know some secret that has eluded researchers in that area.

crest 3y ago

It's a trade-off between availability and correctness. Do you want anything buggy to crash and stay down until the the problem corrected or a supervisor attempt to run the same buggy code again hoping the bug isn't encountered again? There is no one-size-fits-all answer, but just restarting without reporting and counting the error rate is never a good decision.

spullara 3y ago

I think that is the point.

StimDeck 3y ago

[dead]

andrewstuart 3y ago

I design my applications around systemds ability to restart applications quickly and cleanly.