5 ms·

Let's carefully step around the whole distraction that is systemd. Instead: > The real problem here is that it is impossible to diagnose or debug this situatio

Let's carefully step around the whole distraction that is systemd. Instead: > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. This seems to be really common practice in modern software development, and it sucks. There's a rush to rewrite old software and to match some or most of that software's feature set, enough to get people to adopt it, but useful error messages and other diagnostics represent a lot of extra work and so nobody bothers. Not enough programmers are spending time as systems administrators, trying to keep plates spinning, and so they aren't learning to appreciate the value of software that helps you troubleshoot it when something goes wrong. It's turning into the long-running conflict between mechanics and automotive engineers, where mechanics are tired of things like, "To replace [simple part that needs to be replaced every 5 years]: begin unbolting the entire front of the car. The front of the car uses 23 similar, but different, types of fasteners; keep track of them." It needs to be a rule that if somebody's going to rewrite some software, they've got to match or improve the logging and error handling in it. Error handling needs to start getting the kind of attention and shaming for bad behavior that bottom-barrel security practices gets.

gcb0 8y ago
That's not quite right in this case. Systemd and gnome3 were cases where the new 'version' did not need full feature parity. Both those projects were written on leisure time, with no pressure for deadline or feature. and still adopted from top down. i guess some projects forget the benevolent part of benevolent dictator for life. systemd arguments were just "boot faster" and "it's newer". Everyone knew it was going to be broken, buggy, lacking features. As those were never actual goals. In fact, the goals had features never intended for init. So we could say they were rushing for feature anti-parity. https://www.tecmint.com/systemd-replaces-init-in-linux/ https://www.tecmint.com/systemd-replaces-init-in-linux/ see the table at the end. All the features systemd have show "no" for init. And "verbose debug", something you would consider very essential on a piece of software responsible for making your machine something broken and unusable, just show "no" for systemd, and nobody ever cared, because: boot faster (or so they say) systemd is a mistake. But i guess now it is a permanent mistake.
- mikekchar 8y ago
  You're getting down voted mainly because you are wrong. I'm actually not well versed in this, so hopefully somebody will correct my wrong version -- but at least I think it will steer you in the right direction ;-) Systemd solves real problems that have to do with virtualisation and/or containerisation. Red Hat is the company that wrote Systemd and they did it for commercial reasons (i.e. they needed these facilities in order to fulfil contracts with paying customers). Red Hat contributes greatly to Gnome. I haven't checked, but I believe they are far and away the biggest contributor. They have financial reasons for doing so, but I don't really understand what they are. The connection between Systemd and Gnome is real, but not quite what you think. Systemd does make some stuff in Gnome easier to do, but it isn't really necessary at all. Systemd was not developed for Gnome -- it was adopted by Gnome. It's a bit of a dog food problem. Most Gnome developers (i.e. Red Hat developers) have a vested interest in Systemd being successful. They want to make sure they are using it where they can. They saw no particular reason to make Gnome 3 compatible with any other init system because: 1. It would be more work and why should they pay for work that isn't in their interest 2. They want more people to adopt Systemd because that puts Red Hat in a good position to sell more contracts. It's totally understandable and not really underhanded. I personally hate it with a passion, but things I hate are not always evil :-)
  - thwarted 8y ago
    Systemd solves real problems that have to do with virtualisation and/or containerisation. systemd predates the industry love affair with containerization and virtualization for workload management, although work on containerization within systemd may have helped increase awareness of containerization's value. I don't remember early reasons to use systemd mentioning either; the rationales trotted out in favor of systemd in its early days were reducing boot time and replacing shell scripts used during boot, shell scripts labeled things like "old" and "fragile", despite multiple decades of shell scripts being successfully used to reliably manage the boot process on open source and proprietary Unixes, alike.
    - beagle3 8y ago
hvidgaard 8y ago
> > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. > This seems to be really common practice in modern software development, and it sucks I have this discussion with the PM and CEO every now and then. It is impossible to give meaningful error messages for all errors. For known errors you can give a meaningful error message, but for errors not yet discovered it's more luck than anything if you give a meaningful error message. The best you can do it make the software robust so it'll work if a non critical subsystem/plugin fails. The next part of the discussion is usually them suggesting to just allocate time to figure most errors and write meaningful error messages for them. But the problem is that this isn't a mechanical system, but a system several orders of magnitude more complex, and even if we did spend multiple man-years doing it, the cost in terms of added complexity would be enough to get any management directly responsible fired. That said, I completely agree that developers should spend time supporting and running systems so they understand the impact of some of those decisions. Especially should it be expected that when you encounter errors that aren't bugs in the source code, you account for them.
- alxlaz 8y ago
  > It is impossible to give meaningful error messages for all errors. This problem was, however, largely non-existent back when most of the init code was shell code. It was easy to trace, easy to debug. (Edit: obviously, "easy" if you know sh and the idioms. They are, nonetheless, far easier than stracing stuff) It certainly had a lot of other problems (hence systemd's popularity) but debugging was definitely not one of them.
- AnIdiotOnTheNet 8y ago
  If you structure your software simply and make it easy to reason about what is going on, then any half decent error message should give enough information to at least start down the right path. But we don't get that in modern software. We get "Something went wrong :(".
bionoid 8y ago
> This seems to be really common practice in modern software > development, and it sucks. I recently deployed an upgrade of an "enterprise-type" jboss application, and it failed. The log file was literally 200kb of stack traces with some "INFO" entries sprinkled here and there. I sent it to the vendor, and their response was "everything looks okay"
JdeBP 8y ago
I disagree that nobody bothers. I've seen people bother about this.

gcb0 8y ago

That's not quite right in this case. Systemd and gnome3 were cases where the new 'version' did not need full feature parity. Both those projects were written on leisure time, with no pressure for deadline or feature. and still adopted from top down. i guess some projects forget the benevolent part of benevolent dictator for life. systemd arguments were just "boot faster" and "it's newer". Everyone knew it was going to be broken, buggy, lacking features. As those were never actual goals. In fact, the goals had features never intended for init. So we could say they were rushing for feature anti-parity. https://www.tecmint.com/systemd-replaces-init-in-linux/ https://www.tecmint.com/systemd-replaces-init-in-linux/ see the table at the end. All the features systemd have show "no" for init. And "verbose debug", something you would consider very essential on a piece of software responsible for making your machine something broken and unusable, just show "no" for systemd, and nobody ever cared, because: boot faster (or so they say) systemd is a mistake. But i guess now it is a permanent mistake.

mikekchar 8y ago
You're getting down voted mainly because you are wrong. I'm actually not well versed in this, so hopefully somebody will correct my wrong version -- but at least I think it will steer you in the right direction ;-) Systemd solves real problems that have to do with virtualisation and/or containerisation. Red Hat is the company that wrote Systemd and they did it for commercial reasons (i.e. they needed these facilities in order to fulfil contracts with paying customers). Red Hat contributes greatly to Gnome. I haven't checked, but I believe they are far and away the biggest contributor. They have financial reasons for doing so, but I don't really understand what they are. The connection between Systemd and Gnome is real, but not quite what you think. Systemd does make some stuff in Gnome easier to do, but it isn't really necessary at all. Systemd was not developed for Gnome -- it was adopted by Gnome. It's a bit of a dog food problem. Most Gnome developers (i.e. Red Hat developers) have a vested interest in Systemd being successful. They want to make sure they are using it where they can. They saw no particular reason to make Gnome 3 compatible with any other init system because: 1. It would be more work and why should they pay for work that isn't in their interest 2. They want more people to adopt Systemd because that puts Red Hat in a good position to sell more contracts. It's totally understandable and not really underhanded. I personally hate it with a passion, but things I hate are not always evil :-)
- thwarted 8y ago
  Systemd solves real problems that have to do with virtualisation and/or containerisation. systemd predates the industry love affair with containerization and virtualization for workload management, although work on containerization within systemd may have helped increase awareness of containerization's value. I don't remember early reasons to use systemd mentioning either; the rationales trotted out in favor of systemd in its early days were reducing boot time and replacing shell scripts used during boot, shell scripts labeled things like "old" and "fragile", despite multiple decades of shell scripts being successfully used to reliably manage the boot process on open source and proprietary Unixes, alike.
  - beagle3 8y ago

hvidgaard 8y ago

> > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it. > This seems to be really common practice in modern software development, and it sucks I have this discussion with the PM and CEO every now and then. It is impossible to give meaningful error messages for all errors. For known errors you can give a meaningful error message, but for errors not yet discovered it's more luck than anything if you give a meaningful error message. The best you can do it make the software robust so it'll work if a non critical subsystem/plugin fails. The next part of the discussion is usually them suggesting to just allocate time to figure most errors and write meaningful error messages for them. But the problem is that this isn't a mechanical system, but a system several orders of magnitude more complex, and even if we did spend multiple man-years doing it, the cost in terms of added complexity would be enough to get any management directly responsible fired. That said, I completely agree that developers should spend time supporting and running systems so they understand the impact of some of those decisions. Especially should it be expected that when you encounter errors that aren't bugs in the source code, you account for them.

alxlaz 8y ago
> It is impossible to give meaningful error messages for all errors. This problem was, however, largely non-existent back when most of the init code was shell code. It was easy to trace, easy to debug. (Edit: obviously, "easy" if you know sh and the idioms. They are, nonetheless, far easier than stracing stuff) It certainly had a lot of other problems (hence systemd's popularity) but debugging was definitely not one of them.
AnIdiotOnTheNet 8y ago
If you structure your software simply and make it easy to reason about what is going on, then any half decent error message should give enough information to at least start down the right path. But we don't get that in modern software. We get "Something went wrong :(".

bionoid 8y ago

> This seems to be really common practice in modern software > development, and it sucks. I recently deployed an upgrade of an "enterprise-type" jboss application, and it failed. The log file was literally 200kb of stack traces with some "INFO" entries sprinkled here and there. I sent it to the vendor, and their response was "everything looks okay"

JdeBP 8y ago

I disagree that nobody bothers. I've seen people bother about this.