4 ms·

> Why does systemd give up by default? > I’m not sure. If I had to speculate, I would guess the developers wanted to prevent laptops running out of battery too

> Why does systemd give up by default? > I’m not sure. If I had to speculate, I would guess the developers wanted to prevent laptops running out of battery too quickly because one CPU core is permanently busy just restarting some service that’s crashing in a tight loop. sigh … bounded randomized exponential backoff retry. (exponential: double the maximum time you might wait each iteration. Randomized: the time you want is a random amount, between [0, current maximum] (yes, zero.). Bounded: you stop doubling at a certain point, like 5 minutes, so that we'll never wait longer than 5 minutes; otherwise, at some point you're waiting for ∞s, which I guess is like giving up.) (The concern about logs filling up is a worse one. It won't directly solve this, but a high enough max wait usually slows the rate of log generation enough that it becomes small enough to not matter. Also do your log rotations on size.)

deleted 3y ago
[deleted]
kaba0 3y ago
Arguably, this logic should live in another place that monitors the service. Especially that service startup failure is usually not something that gets fixed on its own, like a network connection (where exponential backoff is (in)famous). A bad config file, or a failed disk won’t recover in 10 minutes on its own, so systemd’s default makes sense here, I believe.
- otterley 3y ago
  systemd is a service monitor. It wouldn't be nearly as useful if it wasn't!
- gizmo686 3y ago
  From the servers perspective, external problems typically do get fixed on their own. It is nice when resolving the primary issue is sufficient to fix the entire system; instead of needing to resolve the primary issue; then fix all the secondary and tertiary issues. At my work, we have a simple philosophy for this. The tester is allowed to (on the test system): toggle servers' power; move around network cables; input bad configuration; etc; in any permutation he wants. So long as at the end of the exersise everything is setup correctly the system should function nominally (potentially after a reasonable delay). There should, of course, be a system level dashboard that notifies someone there is a problem; but that is unrelated to the server internal retry logic.
jxf 3y ago
Q: Why is the optimal lower bound zero and not "at least as long as you waited last time"?
- onetoo 3y ago
  edit: I did some more investigation, and I missed something crucial: The distribution of requests over time becomes very wonky if the lower bound isn't 0. It stabilizes given enough time, but that time seems very long. Whereas lower-bound 0 quickly becomes uniform. See the following figure, where I plot the # of requests over type: https://i.imgur.com/PNUFhjc.png https://i.imgur.com/PNUFhjc.png And that is why you should use 0. --- I am not an expert, but I got nerd-sniped. I have made some simulations[1] and done some napkin math[2], and I would summarize as follows: I don't think there is any globally optimal, I think it depends on your exact circumstances, and which of {excess waiting time, excess load} you want to optimize. Defaulting to 0 is a lot easier to implement, and is at most a factor 2 worse than the optimal lower bound w.r.t. waiting time vs. load. You may also consider [0.5 * maximum, maximum], which trades some excess waiting time for less load. Your suggestion is a similar heuristic one might use depending on their exact circumstances. [1] https://gist.github.com/Dryvnt/1984d9389ae7386127f5e8998bf52ea3 https://gist.github.com/Dryvnt/1984d9389ae7386127f5e8998bf52... [2] Consider a family of random bounded exponential back-off strategies with ultimate upper bound U as follows: Strategy X(z): Random of [z * U, U], where z is constant, 0 <= z <= 1 There are other families of back-off algorithms with other characteristics, and I am not considering or comparing those, just this family. Note that strategy OP suggests is X(0). Consider that X(1) is non-random, which is undesirable. Simple probability tells us avg(X(z)) = (1 + z) * U / 2 Server load ~= frequency, frequency is inverse duration, so load(X(z)) ~= 1 / avg(X(z)) = 2 / (1 + z) * U The difference in load between any two X strategies is load(X(z1)) / load(X(z2)) = (1 + z1) / (1 + z2) Since z1 and z2 are constant, this relative load is constant. Due to the bounds on z, the largest possible relative load is load(X(1)) / load(X(0)) = 2 / 1 = 2 Now consider your suggested strategy Strategy Y: Random of [L, U], where L is the last choice Note that Y = X(y), where y = L / U. In fact, Y approaches X(1) exponentially fast, since the difference between L and T is halved each step, on average. So your suggestion still falls within this at-most-factor-2 difference. Exactly where just depends on the outage length.
isatty 3y ago
Regardless, all this opinionated settings should be by OS maintainers or similar. I don’t see why a low level init system tries to make decisions for others. Yes, it may be with good intentions, but don’t.
- bogota 3y ago
  The amount of times i had to fight and debug systemd compared to any other init system is at least 10x. Yes it does a lot of stuff for you and in others I had to write custom scripts but it was much more understandable and maintainable long term. Sadly systemd won and now i build my own OS without it.
  - __float 3y ago
    Even for basic service running, complex dependencies are so much more manageable in systemd. I'm glad systemd "won"; it's much more maintainable IMO than shell scripts written once and forgotten about (until they break).
    - linsomniac 3y ago
      Agreed, I have some friends that don't like systemd, and I respect their opinions greatly, but my experience with systemd has been quite good over the last decade. I really like it much more than the old init.d scripts. With one exception... I tried to set up a inetd-like service where a script was run on TCP connection with the output going out to the socket. It was tricky to get set up, but eventually I got it working... or so I thought. A few days later the system was super sluggish and I found that every connection that had been made to this service was left open and consuming resources. Switched that to an socat running under systemd and it's been fine since. Never could figure out what exactly the deal was.
  - kaba0 3y ago
    With all due respect, I just don’t believe that. Perhaps it’s just rosy glasses on you, or the modern complexity of services/their dependencies.
- 3y ago

deleted 3y ago

[deleted]

kaba0 3y ago

Arguably, this logic should live in another place that monitors the service. Especially that service startup failure is usually not something that gets fixed on its own, like a network connection (where exponential backoff is (in)famous). A bad config file, or a failed disk won’t recover in 10 minutes on its own, so systemd’s default makes sense here, I believe.

otterley 3y ago
systemd is a service monitor. It wouldn't be nearly as useful if it wasn't!
gizmo686 3y ago
From the servers perspective, external problems typically do get fixed on their own. It is nice when resolving the primary issue is sufficient to fix the entire system; instead of needing to resolve the primary issue; then fix all the secondary and tertiary issues. At my work, we have a simple philosophy for this. The tester is allowed to (on the test system): toggle servers' power; move around network cables; input bad configuration; etc; in any permutation he wants. So long as at the end of the exersise everything is setup correctly the system should function nominally (potentially after a reasonable delay). There should, of course, be a system level dashboard that notifies someone there is a problem; but that is unrelated to the server internal retry logic.

jxf 3y ago

Q: Why is the optimal lower bound zero and not "at least as long as you waited last time"?

onetoo 3y ago
edit: I did some more investigation, and I missed something crucial: The distribution of requests over time becomes very wonky if the lower bound isn't 0. It stabilizes given enough time, but that time seems very long. Whereas lower-bound 0 quickly becomes uniform. See the following figure, where I plot the # of requests over type: https://i.imgur.com/PNUFhjc.png https://i.imgur.com/PNUFhjc.png And that is why you should use 0. --- I am not an expert, but I got nerd-sniped. I have made some simulations[1] and done some napkin math[2], and I would summarize as follows: I don't think there is any globally optimal, I think it depends on your exact circumstances, and which of {excess waiting time, excess load} you want to optimize. Defaulting to 0 is a lot easier to implement, and is at most a factor 2 worse than the optimal lower bound w.r.t. waiting time vs. load. You may also consider [0.5 * maximum, maximum], which trades some excess waiting time for less load. Your suggestion is a similar heuristic one might use depending on their exact circumstances. [1] https://gist.github.com/Dryvnt/1984d9389ae7386127f5e8998bf52ea3 https://gist.github.com/Dryvnt/1984d9389ae7386127f5e8998bf52... [2] Consider a family of random bounded exponential back-off strategies with ultimate upper bound U as follows: Strategy X(z): Random of [z * U, U], where z is constant, 0 <= z <= 1 There are other families of back-off algorithms with other characteristics, and I am not considering or comparing those, just this family. Note that strategy OP suggests is X(0). Consider that X(1) is non-random, which is undesirable. Simple probability tells us avg(X(z)) = (1 + z) * U / 2 Server load ~= frequency, frequency is inverse duration, so load(X(z)) ~= 1 / avg(X(z)) = 2 / (1 + z) * U The difference in load between any two X strategies is load(X(z1)) / load(X(z2)) = (1 + z1) / (1 + z2) Since z1 and z2 are constant, this relative load is constant. Due to the bounds on z, the largest possible relative load is load(X(1)) / load(X(0)) = 2 / 1 = 2 Now consider your suggested strategy Strategy Y: Random of [L, U], where L is the last choice Note that Y = X(y), where y = L / U. In fact, Y approaches X(1) exponentially fast, since the difference between L and T is halved each step, on average. So your suggestion still falls within this at-most-factor-2 difference. Exactly where just depends on the outage length.

isatty 3y ago

Regardless, all this opinionated settings should be by OS maintainers or similar. I don’t see why a low level init system tries to make decisions for others. Yes, it may be with good intentions, but don’t.

bogota 3y ago
The amount of times i had to fight and debug systemd compared to any other init system is at least 10x. Yes it does a lot of stuff for you and in others I had to write custom scripts but it was much more understandable and maintainable long term. Sadly systemd won and now i build my own OS without it.
- __float 3y ago
  Even for basic service running, complex dependencies are so much more manageable in systemd. I'm glad systemd "won"; it's much more maintainable IMO than shell scripts written once and forgotten about (until they break).
  - linsomniac 3y ago
    Agreed, I have some friends that don't like systemd, and I respect their opinions greatly, but my experience with systemd has been quite good over the last decade. I really like it much more than the old init.d scripts. With one exception... I tried to set up a inetd-like service where a script was run on TCP connection with the output going out to the socket. It was tricky to get set up, but eventually I got it working... or so I thought. A few days later the system was super sluggish and I found that every connection that had been made to this service was left open and consuming resources. Switched that to an socat running under systemd and it's been fine since. Never could figure out what exactly the deal was.
- kaba0 3y ago
  With all due respect, I just don’t believe that. Perhaps it’s just rosy glasses on you, or the modern complexity of services/their dependencies.
3y ago