3 ms·

yes it's still that bad, my previously reliable systems are now non-deterministic and break randomly every few days I discover some new bit of non-determinism:

yes it's still that bad, my previously reliable systems are now non-deterministic and break randomly every few days I discover some new bit of non-determinism: - systems getting stuck for long periods at startup or reboot with with some useless text like "Waiting for user session to shutdown" - reboots never completing at all (getting completely wedged shutting down) - jobs starting in weird orders, like crond starting up before the network, causing various things to break in new novel ways (sometimes, not always) I can fix most of these systemd-created issues one by one as I discover them, but it's hardly making my life easier

telmich 7y ago
I have the same getting stuck problems plus resolving issues as soon as systemd-resolved is in the game
bengerbil 7y ago
I think some of my pain came from attempts to unmount everything (particularly NFS) before letting long-running shutdowns (graphite/carbon) finished flushing, then never trying again. While diving into that and eventually coming up with a sufficient set of dependencies, I would fondly think back to the two (or more) phases of unmounting at shutdown that had that solved cold many years prior.
acdha 7y ago
All of those issues happened with SysV init, as well — there are literally decades of history of people working around things like unmounting failed filesystems, watchdogs to kill -9 hung daemons, etc. For example, you mentioned crond startup before the network but that's hardly new with systemd — checking the oldest system I have access to shows that it's configured with two dependencies: “# Required-Start: $local_fs $syslog”. To the extent that systemd changed anything it was exposing that an assumption was wrong — and the nice thing is that you can use a systemd service to replace that cron task with something which _does_ behave the way you thought with precise service dependencies and all of the other benefits to using systemd (e.g. resource management): https://www.freedesktop.org/software/systemd/man/systemd.timer.html https://www.freedesktop.org/software/systemd/man/systemd.tim...
- blibble 7y ago
  it's the non-determinism that's the problem as I said, I can fix the issues one by one as I notice them, but if they only occur 1 boot in 10, or maybe 1 in 100 it's miserable and a much less reliable system than before but hey, the systems now boot 2 seconds faster (if they boot at all...)
  - acdha 7y ago
    Again, you already had a problem. It might not have been as obvious because it was enough slower to avoid the race condition under normal circumstances but I’ve hit those problems many times over the years.
    - blibble 7y ago
      Again, I've said that I know there are issues, that I end up discovering one at a time. the problem is not the issues themselves, as once I know they're there I can trivially google for how to fix them, it's the fact they don't show up except sometimes I do not care about boot speed at all, I care that I upgrade the system to a new distro release with systemd, test it, then it continues to work that way until I change something else this is no longer the case, and in 20 odd years of maintaining systems I've never had as much "fun" before as I've had over the last couple of years with systemd I just want my systems to boot, run my simple applications, then shutdown when I ask them to
      - acdha 7y ago
        > I just want my systems to boot, run my simple applications, then shutdown when I ask them to Which is what you get out of the box with systemd. The only problems you mentioned either apply equally (e.g. shutdown delays due to blocked processes — having supported NFS clients, let me tell you that was not a new problem in the 90s…) or are due to misunderstandings about how to use other components. For example, if you have a critical service started in cron with no error handling or recovery, you always had an unstable system and were simply lucky not to have seen it fail before now. Projects like Upstart and systemd were started in large part due to people getting tired of having to kludge around those limitations of the older approaches. Having fixed systems which were configured that way in the past, I much prefer using something which solves it by design than continuing to bet on being lucky.

telmich 7y ago

I have the same getting stuck problems plus resolving issues as soon as systemd-resolved is in the game

bengerbil 7y ago

I think some of my pain came from attempts to unmount everything (particularly NFS) before letting long-running shutdowns (graphite/carbon) finished flushing, then never trying again. While diving into that and eventually coming up with a sufficient set of dependencies, I would fondly think back to the two (or more) phases of unmounting at shutdown that had that solved cold many years prior.

acdha 7y ago

All of those issues happened with SysV init, as well — there are literally decades of history of people working around things like unmounting failed filesystems, watchdogs to kill -9 hung daemons, etc. For example, you mentioned crond startup before the network but that's hardly new with systemd — checking the oldest system I have access to shows that it's configured with two dependencies: “# Required-Start: $local_fs $syslog”. To the extent that systemd changed anything it was exposing that an assumption was wrong — and the nice thing is that you can use a systemd service to replace that cron task with something which _does_ behave the way you thought with precise service dependencies and all of the other benefits to using systemd (e.g. resource management): https://www.freedesktop.org/software/systemd/man/systemd.timer.html https://www.freedesktop.org/software/systemd/man/systemd.tim...

blibble 7y ago
it's the non-determinism that's the problem as I said, I can fix the issues one by one as I notice them, but if they only occur 1 boot in 10, or maybe 1 in 100 it's miserable and a much less reliable system than before but hey, the systems now boot 2 seconds faster (if they boot at all...)
- acdha 7y ago
  Again, you already had a problem. It might not have been as obvious because it was enough slower to avoid the race condition under normal circumstances but I’ve hit those problems many times over the years.
  - blibble 7y ago
    Again, I've said that I know there are issues, that I end up discovering one at a time. the problem is not the issues themselves, as once I know they're there I can trivially google for how to fix them, it's the fact they don't show up except sometimes I do not care about boot speed at all, I care that I upgrade the system to a new distro release with systemd, test it, then it continues to work that way until I change something else this is no longer the case, and in 20 odd years of maintaining systems I've never had as much "fun" before as I've had over the last couple of years with systemd I just want my systems to boot, run my simple applications, then shutdown when I ask them to
    - acdha 7y ago
      > I just want my systems to boot, run my simple applications, then shutdown when I ask them to Which is what you get out of the box with systemd. The only problems you mentioned either apply equally (e.g. shutdown delays due to blocked processes — having supported NFS clients, let me tell you that was not a new problem in the 90s…) or are due to misunderstandings about how to use other components. For example, if you have a critical service started in cron with no error handling or recovery, you always had an unstable system and were simply lucky not to have seen it fail before now. Projects like Upstart and systemd were started in large part due to people getting tired of having to kludge around those limitations of the older approaches. Having fixed systems which were configured that way in the past, I much prefer using something which solves it by design than continuing to bet on being lucky.