10 ms·

I had no opinion on systemd until yesterday. In fact I had a glace or two at the code and it's pretty clean and I liked the rough objectives laid out. I instal

I had no opinion on systemd until yesterday. In fact I had a glace or two at the code and it's pretty clean and I liked the rough objectives laid out. I installed CentOS 7 on a machine last night that we're replacing CentOS 6 on and was poked in the face with timedatectl and dbus problems for an entire hour, some of which were intermittent. Debugging these issues is a horrific pain. I lost 4 hours on it. I've never lost that much time on a system function before. This is not what I expected and there is no way I could possibly introduce that to our production environment. I think that might why people are slightly sensitive to it. Yes you're exactly right, but replacing something with something less stable, more complicated and more difficult to debug isn't a rational or good engineering. I'm sure many people will be fed up with systemd much quicker than what was already there. Not impressed with a community which pushes this as stable, quality software. Voting with my feet: FreeBSD is being trialled instead. WhatsApp throwing a million dollars at it draws a lot of valuable attention and puts it in the business's mindset. Choice is as much of a valuable aspect of open source too...

paulfurtado 12y ago
Out of curiosity, what ended up being the root cause of the timedatectl problem?
- badgersandjam 12y ago
  Absolutely no idea. It just went away spontaneously which is even more worrying as that suggests the system is non-deterministic. I don't have the error on my phone which I'm on at the moment but it threw a dbus error with no debug info.
  - watt 12y ago
    It must be a hardware problem on your side, dude. Simple as that. I imagine 10 years ago you would be the person complaining that GCC segfaults randomly during compiling Linux kernel, complaining that it's not "tested completely". While the segfault was caused by CPU overheating (not cooled properly) and flaky memory (causing bits to flip).
    - badgersandjam 12y ago
      Known good HP DL380 G8 pulled from production, ECC RAM, monitoring, SAS disks, full hardware test and memtest86 pass. Nope. Not that. We don't buy crappy hardware or not test it.
    - dap 12y ago
      Dude, prove it. The system should be able to exonerate itself by detecting and reporting those problems (and other systems do). At the very least, point to some actual evidence. This is the "engineering" process that the parent poster referred to elsewhere. Just because a problem is unusual, intermittent, or only affects one person doesn't mean it's not a regular old software bug. And in my experience, it almost always is. And once you do debug it, you often (but not always) understand why it was intermittent, under what conditions it happened, and why you were the only person that saw it.
  - ambrice 12y ago
    If you don't know what the problem was why are you so convinced it was systemd? Could be the kernel, could be udev, dbus, filesystem, or hardware (sorry, there is never 100% "known good" hardware).
- deleted 12y ago
  [deleted]
gtirloni 12y ago
It hardly does anything for FreeBSD in the enterprise world. Companies cannot afford to support Windows Server, Linux AND another flavor of UNIX (FreeBSD). They are already dumping Solaris/AIX/HP-UX as much as they can so environments are a bit more homogeneous (and easier to support). There is no point in onboarding FreeBSD for what they perceive as minor technical differences (that, truth be told, are overshadowed by dozens of layers of bureaucracy and any efficiency benefit is completely lost). It took a long time and a giant ecosystem to get where Linux is today at big enterprises. OSes are commodities in that space. They are not commodities in many other spaces though (e.g. startups, HPC, science, etc).
- badgersandjam 12y ago
  I disagree. There are very few heterogenic systems in the enterprise. That is an objective, but the main thing is that we deliver what we're paid to deliver by choosing an appropriate platform. We have Solaris, zSeries, Linux and Windows. We just got rid of AIX. As for minor differences, FreeBSD has a lot of much bigger wins than people realise at first glimpse. The differences are far from minor. For example: ZFS, dtrace, rctl, a scary good IP stack, virtio support, documentation that doesn't suck, a POSIX base, LLVM/clang, a MAC framework that doesn't suck, OpenBSM, CARP and a pile more. Oh plus an automated deployment story that is pretty tidy. Sure we can replicate some of these on CentOS 7 for example with similar tech but the above are a million times more cohesive.
- feld 12y ago
  > They are already dumping Solaris/AIX/HP-UX as much as they can so environments are a bit more homogeneous (and easier to support). Whoever is pushing for this is an idiot then. Verisign, for example, has had 100% DNS uptime for the .net, .org, and .com root servers for ~15 years because of their mixed environments. In every one of their POPs they tend to have at least two racks of equipment with: * 2 different brands of load balancers * 2 different brands of firewalls * 2 different brands of switches * 2 different brands of servers ** servers are from different hardware generations * 2 different OSes (Linux and FreeBSD) * a choice from 3 different DNS server software This is all pretty much randomized at each location. As a result, a bug in one piece of the stack (hardware, software, driver, security, etc) will not take down their service completely. This is how you run a reliable global-scale service. Anyone who plays the "it's just easier if we all use ____" is in for a big surprise when their entire infrastructure is at risk due to one bug.
olau 12y ago
Are you really extrapolating from ONE data point? In some sense systemd is more stable in that it's fixing some longstanding bugs with sysvinit, but of course it will have some bugs of it own. If you don't want to deal with that, you could skip a release.
- badgersandjam 12y ago
  No it's not just one data point. This is the final straw to use the old phrase. After a few years of serious problems, our most recent being CIFS VFS problems causing panics and mounts locking up on CentOS 6.5, hard locks on RH certified hardware, power management hell and so much incredible churn with no progress and the sudden "fuck POSIX" approach, it paints a really bad picture of the current state of things. There is a distinct lack of engineering prowess and quality control. It originates at the core GNU + kernel + freedesktop teams and waterfalls down through the distribution houses. That's the problem and it's endemic within Linux.
  - moron4hire 12y ago
    I caution you to not apply the word "engineering" to software, especially in the apparent manner you've been applying it to this thread. You seem to have an idea that there are universal software development practices that are so well understood we can make regulation around them. But that isn't how software works. There are some practices that are known and some that are unknown. For working software "engineers", we are all, daily, put into a position for having to figure out for ourselves even what tools and materials to use to make anything. Imagine you were an architect of buildings. Your day to day job is to design mundane strip malls and gas stations. You have building code on your side for much of the process. As long as you don't violate the regulation, you at worst can only make an inconvenient building, but not a dangerous one. But imagine instead that you're building a large office building every six months, your clients demand you don't reuse any design principles on your future clients, and you not only lacked the building code, but also 1/4th of the heavy machine equipment, 1/2ths of the tools, and 3/4ths of the raw materials. I don't just mean you don't have them in stock, I mean nobody has invented them yet. And of the ones that have been invented, we don't even know all of their material properties, say nothing about what material properties we should be looking for. Will this particular bolt we are using with these particular cross beams hold up to the stresses placed on them? The answer is unknown, and I'm some cases unknowable. It's really easy as a user to say, "they should have tested this more". While strictly true that more testing may have found your issues ahead of time (presuming the right tests were done), it is inefficient engineering to exhaustively test things. Even mechanical engineering bases a lot on statistical modeling, which will always, always have corner cases that don't match reality. In the real world, people had to learn the hard way about things like lightning rods and sacrificial electrodes. They didn't come about from "testing during development". They came about from testing live, and seeing which buildings and boats did or did not burn down or sink. That's not bad engineering. That's just the nature of unknown problems.
nailer 12y ago
I installed CentOS 7, with no prior systemd experience, had no problems and found .service files way neater than sysvinit bash scripts and liked having meaningful names in the log/ journal rather than 'local3'.
- badgersandjam 12y ago
  I had no problems the second time as well. Unfortunately when the first and second time differ even though identical (recorded) steps were performed, one has to ask the question: why and can I trust it?
  - rantanplan 12y ago
    Maybe it was a hardware problem on your end? My rule of thumb is "search for the problem on Google. If nothing comes up, maybe something is wrong on my end". Did you find any results or reported bugs similar to what you experienced?
    - badgersandjam 12y ago
      The hardware is known good on CentOS 6.5. In fact it was high end HP kit that we pulled out of production. Yes there were other mentions of it with notes to it being fixed in a later systemd drop, which we can't deploy because RH/CentOS don't ship it. I think one of our guys raised a case with RH but I was dragged off onto something else then.
      - e12e 12y ago
        To be fair, pulseaudio did expose bugs in alsa-drivers, so a "known good" configuration could stop working when "upgraded" to use PA. I have my share of reservations about systemd (and PA), but thought that it might be worth pointing out that "known good" hardware A with software X, doesn't have to mean hardware A is all good, just that A has no bugs/errors not exposed when running X. So Y comes along (new kernel, drivers?) with entirely new code - and suddenly things behave erratically.
billybofh 12y ago
I've been a longtime linux sysadmin (back into the 90's) and have run into similar issues as you. I've never had as many problems with basic system stuff as I have since testing some of the new systemd-defaulting distro's. For instance, the centos/rhel 7 boxes I've tried have erratic problems - sometimes not setting the hostname, sometimes services don't start, sometimes services I've disabled _do_ start. It's making me really think about shifting to FreeBSD (or, god help me back to Solaris).
seivadmas 12y ago
You know this is funny, I remember reading comments EXACTLY like this about 3-4 years ago but with pulseaudio in place of systemd. Pulseaudio was Lennart's previous project. It broke everything in linux sound for a while, everybody moaned and hated it and said it was the worst thing since the crucifixion of Christ. Yet, name one problem you had with sound on linux in the past year? There are very few. Pulseaudio now just works(tm) and is a unseed, unheard of part of the plumbing. If you remember what is was like messing with ALSA and (shudders) OSS before pulseaudio came along you will agree that the current state of affairs is a million miles better. It used to be really difficult to get more than one application to be able to play sound at a time. I remember compiling sound drivers from source just to get them working. Configuring ALSA config files to get surround sound working was practically a black art. Creating manual scripts that unmute the sound card on every boot because the driver didn't initialize it properly. With pulseaudio, I never have to worry about any of that and configuring surround sound takes me two clicks of the mouse. Lennart did a fantastic job with pulseaudio, he took on a dirty problem that nobody else dared to touch and went through years of criticism to produce a really high quality solution that solved the linux audio problem so well that you don't hear complaints about it anymore. In light of that, I trust him to do a good job with systemd. It'll be a couple of years of everyone moaning and bitching and whining about it, then one day it will have become a seamless part of the plumbing, everyone will take it for granted and wonder how they ever managed fighting with shell scripts and fragmented init systems before systemd came along. It's ironic that Lennart Poettering is probably the most abused developer in the entire OSS ecosystem, yet he is one of the people contributing most to it. For our sake, I'm glad he has such a thick skin. If I was him I'd have quit this game long ago.
- aidanhs 12y ago
  Let's say PulseAudio is really good now (I can't disagree). It was initially released 10 years ago. So 6-7 years after initial release ("3-4 years ago") it was causing people grief. I'm not sure that's a great vote of confidence for the road ahead of systemd (given that systemd presumably has a bit more to it than PulseAudio). To quote the article, "I do honestly believe this will end up being the start of a rocky period for Linux". Ubuntu 14.04 will keep me happy for ~5 years, then I can take another look at what the current state of things are.

paulfurtado 12y ago

Out of curiosity, what ended up being the root cause of the timedatectl problem?

badgersandjam 12y ago
Absolutely no idea. It just went away spontaneously which is even more worrying as that suggests the system is non-deterministic. I don't have the error on my phone which I'm on at the moment but it threw a dbus error with no debug info.
- watt 12y ago
  It must be a hardware problem on your side, dude. Simple as that. I imagine 10 years ago you would be the person complaining that GCC segfaults randomly during compiling Linux kernel, complaining that it's not "tested completely". While the segfault was caused by CPU overheating (not cooled properly) and flaky memory (causing bits to flip).
  - badgersandjam 12y ago
    Known good HP DL380 G8 pulled from production, ECC RAM, monitoring, SAS disks, full hardware test and memtest86 pass. Nope. Not that. We don't buy crappy hardware or not test it.
  - dap 12y ago
    Dude, prove it. The system should be able to exonerate itself by detecting and reporting those problems (and other systems do). At the very least, point to some actual evidence. This is the "engineering" process that the parent poster referred to elsewhere. Just because a problem is unusual, intermittent, or only affects one person doesn't mean it's not a regular old software bug. And in my experience, it almost always is. And once you do debug it, you often (but not always) understand why it was intermittent, under what conditions it happened, and why you were the only person that saw it.
- ambrice 12y ago
  If you don't know what the problem was why are you so convinced it was systemd? Could be the kernel, could be udev, dbus, filesystem, or hardware (sorry, there is never 100% "known good" hardware).
deleted 12y ago
[deleted]

gtirloni 12y ago

It hardly does anything for FreeBSD in the enterprise world. Companies cannot afford to support Windows Server, Linux AND another flavor of UNIX (FreeBSD). They are already dumping Solaris/AIX/HP-UX as much as they can so environments are a bit more homogeneous (and easier to support). There is no point in onboarding FreeBSD for what they perceive as minor technical differences (that, truth be told, are overshadowed by dozens of layers of bureaucracy and any efficiency benefit is completely lost). It took a long time and a giant ecosystem to get where Linux is today at big enterprises. OSes are commodities in that space. They are not commodities in many other spaces though (e.g. startups, HPC, science, etc).

badgersandjam 12y ago
I disagree. There are very few heterogenic systems in the enterprise. That is an objective, but the main thing is that we deliver what we're paid to deliver by choosing an appropriate platform. We have Solaris, zSeries, Linux and Windows. We just got rid of AIX. As for minor differences, FreeBSD has a lot of much bigger wins than people realise at first glimpse. The differences are far from minor. For example: ZFS, dtrace, rctl, a scary good IP stack, virtio support, documentation that doesn't suck, a POSIX base, LLVM/clang, a MAC framework that doesn't suck, OpenBSM, CARP and a pile more. Oh plus an automated deployment story that is pretty tidy. Sure we can replicate some of these on CentOS 7 for example with similar tech but the above are a million times more cohesive.
feld 12y ago
> They are already dumping Solaris/AIX/HP-UX as much as they can so environments are a bit more homogeneous (and easier to support). Whoever is pushing for this is an idiot then. Verisign, for example, has had 100% DNS uptime for the .net, .org, and .com root servers for ~15 years because of their mixed environments. In every one of their POPs they tend to have at least two racks of equipment with: * 2 different brands of load balancers * 2 different brands of firewalls * 2 different brands of switches * 2 different brands of servers ** servers are from different hardware generations * 2 different OSes (Linux and FreeBSD) * a choice from 3 different DNS server software This is all pretty much randomized at each location. As a result, a bug in one piece of the stack (hardware, software, driver, security, etc) will not take down their service completely. This is how you run a reliable global-scale service. Anyone who plays the "it's just easier if we all use ____" is in for a big surprise when their entire infrastructure is at risk due to one bug.

olau 12y ago

Are you really extrapolating from ONE data point? In some sense systemd is more stable in that it's fixing some longstanding bugs with sysvinit, but of course it will have some bugs of it own. If you don't want to deal with that, you could skip a release.

badgersandjam 12y ago
No it's not just one data point. This is the final straw to use the old phrase. After a few years of serious problems, our most recent being CIFS VFS problems causing panics and mounts locking up on CentOS 6.5, hard locks on RH certified hardware, power management hell and so much incredible churn with no progress and the sudden "fuck POSIX" approach, it paints a really bad picture of the current state of things. There is a distinct lack of engineering prowess and quality control. It originates at the core GNU + kernel + freedesktop teams and waterfalls down through the distribution houses. That's the problem and it's endemic within Linux.
- moron4hire 12y ago
  I caution you to not apply the word "engineering" to software, especially in the apparent manner you've been applying it to this thread. You seem to have an idea that there are universal software development practices that are so well understood we can make regulation around them. But that isn't how software works. There are some practices that are known and some that are unknown. For working software "engineers", we are all, daily, put into a position for having to figure out for ourselves even what tools and materials to use to make anything. Imagine you were an architect of buildings. Your day to day job is to design mundane strip malls and gas stations. You have building code on your side for much of the process. As long as you don't violate the regulation, you at worst can only make an inconvenient building, but not a dangerous one. But imagine instead that you're building a large office building every six months, your clients demand you don't reuse any design principles on your future clients, and you not only lacked the building code, but also 1/4th of the heavy machine equipment, 1/2ths of the tools, and 3/4ths of the raw materials. I don't just mean you don't have them in stock, I mean nobody has invented them yet. And of the ones that have been invented, we don't even know all of their material properties, say nothing about what material properties we should be looking for. Will this particular bolt we are using with these particular cross beams hold up to the stresses placed on them? The answer is unknown, and I'm some cases unknowable. It's really easy as a user to say, "they should have tested this more". While strictly true that more testing may have found your issues ahead of time (presuming the right tests were done), it is inefficient engineering to exhaustively test things. Even mechanical engineering bases a lot on statistical modeling, which will always, always have corner cases that don't match reality. In the real world, people had to learn the hard way about things like lightning rods and sacrificial electrodes. They didn't come about from "testing during development". They came about from testing live, and seeing which buildings and boats did or did not burn down or sink. That's not bad engineering. That's just the nature of unknown problems.

nailer 12y ago

I installed CentOS 7, with no prior systemd experience, had no problems and found .service files way neater than sysvinit bash scripts and liked having meaningful names in the log/ journal rather than 'local3'.

badgersandjam 12y ago
I had no problems the second time as well. Unfortunately when the first and second time differ even though identical (recorded) steps were performed, one has to ask the question: why and can I trust it?
- rantanplan 12y ago
  Maybe it was a hardware problem on your end? My rule of thumb is "search for the problem on Google. If nothing comes up, maybe something is wrong on my end". Did you find any results or reported bugs similar to what you experienced?
  - badgersandjam 12y ago
    The hardware is known good on CentOS 6.5. In fact it was high end HP kit that we pulled out of production. Yes there were other mentions of it with notes to it being fixed in a later systemd drop, which we can't deploy because RH/CentOS don't ship it. I think one of our guys raised a case with RH but I was dragged off onto something else then.
    - e12e 12y ago
      To be fair, pulseaudio did expose bugs in alsa-drivers, so a "known good" configuration could stop working when "upgraded" to use PA. I have my share of reservations about systemd (and PA), but thought that it might be worth pointing out that "known good" hardware A with software X, doesn't have to mean hardware A is all good, just that A has no bugs/errors not exposed when running X. So Y comes along (new kernel, drivers?) with entirely new code - and suddenly things behave erratically.

billybofh 12y ago

I've been a longtime linux sysadmin (back into the 90's) and have run into similar issues as you. I've never had as many problems with basic system stuff as I have since testing some of the new systemd-defaulting distro's. For instance, the centos/rhel 7 boxes I've tried have erratic problems - sometimes not setting the hostname, sometimes services don't start, sometimes services I've disabled _do_ start. It's making me really think about shifting to FreeBSD (or, god help me back to Solaris).

seivadmas 12y ago

You know this is funny, I remember reading comments EXACTLY like this about 3-4 years ago but with pulseaudio in place of systemd. Pulseaudio was Lennart's previous project. It broke everything in linux sound for a while, everybody moaned and hated it and said it was the worst thing since the crucifixion of Christ. Yet, name one problem you had with sound on linux in the past year? There are very few. Pulseaudio now just works(tm) and is a unseed, unheard of part of the plumbing. If you remember what is was like messing with ALSA and (shudders) OSS before pulseaudio came along you will agree that the current state of affairs is a million miles better. It used to be really difficult to get more than one application to be able to play sound at a time. I remember compiling sound drivers from source just to get them working. Configuring ALSA config files to get surround sound working was practically a black art. Creating manual scripts that unmute the sound card on every boot because the driver didn't initialize it properly. With pulseaudio, I never have to worry about any of that and configuring surround sound takes me two clicks of the mouse. Lennart did a fantastic job with pulseaudio, he took on a dirty problem that nobody else dared to touch and went through years of criticism to produce a really high quality solution that solved the linux audio problem so well that you don't hear complaints about it anymore. In light of that, I trust him to do a good job with systemd. It'll be a couple of years of everyone moaning and bitching and whining about it, then one day it will have become a seamless part of the plumbing, everyone will take it for granted and wonder how they ever managed fighting with shell scripts and fragmented init systems before systemd came along. It's ironic that Lennart Poettering is probably the most abused developer in the entire OSS ecosystem, yet he is one of the people contributing most to it. For our sake, I'm glad he has such a thick skin. If I was him I'd have quit this game long ago.

aidanhs 12y ago
Let's say PulseAudio is really good now (I can't disagree). It was initially released 10 years ago. So 6-7 years after initial release ("3-4 years ago") it was causing people grief. I'm not sure that's a great vote of confidence for the road ahead of systemd (given that systemd presumably has a bit more to it than PulseAudio). To quote the article, "I do honestly believe this will end up being the start of a rocky period for Linux". Ubuntu 14.04 will keep me happy for ~5 years, then I can take another look at what the current state of things are.