4 ms·

Surprised nobody's mentioned systemd yet! My worst experience was when I was on a cross-country train writing code for a project whose database was mysql. No i

by chaps 7y ago

Surprised nobody's mentioned systemd yet! My worst experience was when I was on a cross-country train writing code for a project whose database was mysql. No internet, intentionally. Something happened with one of my inserts (can't remember what exactly) and I had to restart mysqld. Problem was, with the bad insert, mysqld took longer than 30s to start, which forced systemd to restart it because of a default, undocumented setting to restart a service if it's not up after 30s. Again. And again. And again. Nothing in the documentation hinted that this was a feature of systemd anywhere, and I was stuck troubleshooting that nonsense instead of just writing code. Systemd is just filled with undocumented crap and features that shouldn't exist by default, leading to headache after headache after headache. Such a pain in the ass.

throwaway_se099 7y ago
> Nothing in the documentation hinted that this was a feature of systemd anywhere Nothing except three paragraphs in systemd.service(5) (TimeoutStartSec=, TimeoutStopSec=, TimeoutSec=), with references to systemd-system.conf(5) where the defaults are described. I wouldn't call that undocumented. Whether it should exist by default is another question; on balance, I'm glad it does.
- chaps 7y ago
  This seems like the bug I was having at the time: https://github.com/systemd/systemd/issues/3912 https://github.com/systemd/systemd/issues/3912 Or this: https://unix.stackexchange.com/questions/227017/how-to-change-systemd-service-timeout-value https://unix.stackexchange.com/questions/227017/how-to-chang... Or this: https://github.com/systemd/systemd/issues/5773 https://github.com/systemd/systemd/issues/5773 Or: https://github.com/systemd/systemd/issues/3901 https://github.com/systemd/systemd/issues/3901 Or: https://github.com/systemd/systemd/issues/2047 https://github.com/systemd/systemd/issues/2047 I don't remember seeing any documentation at the time, but I'm clearly not the only one who's had issues with this exact problem. The last one has recent examples of people not being able to find it, and understandably so. Their documentation is shit, fullstop, and no reasonable service should restart in a loop by default after 90s. That's just asinine.
  - throwaway_se099 7y ago
    This seems to be veering in the direction of the Land of Moved Goalposts and Non-Sequiturs, but I'm game for one more round, because lockdown. It's rather unlikely that the first issue, which is an ordering problem during service shutdown, has much in common with timeouts during service startup, which you have described as your original issue. The Stack Exchange question is a straightforward how-to for adjusting the timeouts. 5773 requests a clarification of the documentation of timeouts and signalling behavior. 3901 is a case of misplacing the timeout directive. 2047 is a sort of philosophical problem of reporting uniformity vs. consistency with manual configuration. The systemd team prefers the former, whereas I would prefer the latter for not violating the principle of least surprise, although I understand the reasons for choosing the first option. Anyhow, I still don't see good support for your assertion of the complete lack of timeout documentation, just tangentially related distractions. A complex set of problems, and service management does qualify, will have edge cases, insufficiently clear documentation, and outright bugs, which is why software is never finished. Older init systems had all of these, but they didn't give you the tools to attempt to rectify the problems, so it was all more or less swept under the rug. Using a facile "shit, fullstop" characterization therefore tells me more about your views than the actual technical merits or documentation quality of systemd.
    - chaps 7y ago
      The post is about how systemd wasted lots of time, in a thread about software that has wasted lots of time. The issues around bad documentation and the specific issue (the last one is probably the issue I had, but this was five years ago lol. Knowing me and arch linux, I probably didn't have the full man pages installed, so the cli tools were the only thing I had available.) wasted me a lot of time and very clearly others, and that's exactly the point of this discussion! Hell, to troubleshoot that issue, I even ran something like a `find / -exec strings -a '{}' ; | grep timeout` and found nothing that ended up being helpful. But I absolutely had some form of documentation, again likely the last link I provided which isn't simply a philosophical issue when it provides misleading information that would have ultimately fixed the issue if it gave more informative output! You admit yourself that the documentation can be insufficiently clear with bugs, and it's true: systemd happens to have lots of time killing bugs. The whole point of this thread. You're missing the forest for the trees, friend.
tetris11 7y ago
Never write new code on a train. I've learned this lesson many times. The WiFi is not reliable, the offline documentation is spotty (with the exception of emacs), and you'll waste more time than needed trying to do things the wrong way instead of searching online for solutions later. No new code on the train. Optimise existing or fill in some TODOs. No new code on the train.
- chaps 7y ago
  Nah, I'll keep doing it. Thanks for the suggestion, though.

by chaps 7y ago

throwaway_se099 7y ago

> Nothing in the documentation hinted that this was a feature of systemd anywhere Nothing except three paragraphs in systemd.service(5) (TimeoutStartSec=, TimeoutStopSec=, TimeoutSec=), with references to systemd-system.conf(5) where the defaults are described. I wouldn't call that undocumented. Whether it should exist by default is another question; on balance, I'm glad it does.

chaps 7y ago
This seems like the bug I was having at the time: https://github.com/systemd/systemd/issues/3912 https://github.com/systemd/systemd/issues/3912 Or this: https://unix.stackexchange.com/questions/227017/how-to-change-systemd-service-timeout-value https://unix.stackexchange.com/questions/227017/how-to-chang... Or this: https://github.com/systemd/systemd/issues/5773 https://github.com/systemd/systemd/issues/5773 Or: https://github.com/systemd/systemd/issues/3901 https://github.com/systemd/systemd/issues/3901 Or: https://github.com/systemd/systemd/issues/2047 https://github.com/systemd/systemd/issues/2047 I don't remember seeing any documentation at the time, but I'm clearly not the only one who's had issues with this exact problem. The last one has recent examples of people not being able to find it, and understandably so. Their documentation is shit, fullstop, and no reasonable service should restart in a loop by default after 90s. That's just asinine.
- throwaway_se099 7y ago
  This seems to be veering in the direction of the Land of Moved Goalposts and Non-Sequiturs, but I'm game for one more round, because lockdown. It's rather unlikely that the first issue, which is an ordering problem during service shutdown, has much in common with timeouts during service startup, which you have described as your original issue. The Stack Exchange question is a straightforward how-to for adjusting the timeouts. 5773 requests a clarification of the documentation of timeouts and signalling behavior. 3901 is a case of misplacing the timeout directive. 2047 is a sort of philosophical problem of reporting uniformity vs. consistency with manual configuration. The systemd team prefers the former, whereas I would prefer the latter for not violating the principle of least surprise, although I understand the reasons for choosing the first option. Anyhow, I still don't see good support for your assertion of the complete lack of timeout documentation, just tangentially related distractions. A complex set of problems, and service management does qualify, will have edge cases, insufficiently clear documentation, and outright bugs, which is why software is never finished. Older init systems had all of these, but they didn't give you the tools to attempt to rectify the problems, so it was all more or less swept under the rug. Using a facile "shit, fullstop" characterization therefore tells me more about your views than the actual technical merits or documentation quality of systemd.
  - chaps 7y ago
    The post is about how systemd wasted lots of time, in a thread about software that has wasted lots of time. The issues around bad documentation and the specific issue (the last one is probably the issue I had, but this was five years ago lol. Knowing me and arch linux, I probably didn't have the full man pages installed, so the cli tools were the only thing I had available.) wasted me a lot of time and very clearly others, and that's exactly the point of this discussion! Hell, to troubleshoot that issue, I even ran something like a `find / -exec strings -a '{}' ; | grep timeout` and found nothing that ended up being helpful. But I absolutely had some form of documentation, again likely the last link I provided which isn't simply a philosophical issue when it provides misleading information that would have ultimately fixed the issue if it gave more informative output! You admit yourself that the documentation can be insufficiently clear with bugs, and it's true: systemd happens to have lots of time killing bugs. The whole point of this thread. You're missing the forest for the trees, friend.

tetris11 7y ago

Never write new code on a train. I've learned this lesson many times. The WiFi is not reliable, the offline documentation is spotty (with the exception of emacs), and you'll waste more time than needed trying to do things the wrong way instead of searching online for solutions later. No new code on the train. Optimise existing or fill in some TODOs. No new code on the train.

chaps 7y ago
Nah, I'll keep doing it. Thanks for the suggestion, though.