7 ms·

The policy of ‘oopsing’ and limping on is, in my opinion, literally one of Linux’s worst features. It has bitten me in various cases: - Remember when Linux had

The policy of ‘oopsing’ and limping on is, in my opinion, literally one of Linux’s worst features. It has bitten me in various cases: - Remember when Linux had that caused the kernel to partially crash and eat 100% CPU due to some bug in the leap second application code? That caused a >1MW spike in power usage at Hetzner at the time. That must have been >1GW globally. Many people didn’t notice it immediately, so it must have taken weeks before everyone rebooted. - I’ve personally run into issues where not crashing caused Linux to go on and eat my file system. On any Linux server I maintain, I always toggle those sysctls that cause the kernel to panic on oops, and reboot on panic.

xyzzy_plugh 4y ago
Right, the fact that you can toggle those sysctls is sort of the point. There are plenty of environments where running at all is better than not, hardware watchdogs can solve for unresponsiveness. TFA is about making it possible for the kernel to decide what to do, rather than exploding on the spot, which is terrible.
deleted 4y ago
[deleted]
mike_hock 4y ago
So instead of a power spike, we'd have had a major internet outage across the world, across the entire industry and beyond, probably, if everyone had panicked on oops. The blame really lies with people not monitoring their systems. As you said, you have the option to reboot on panic, but Linus is absolutely not wrong that this size does not fit all. What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? Hitting an assert in those kinds of systems is a very bad place to be, but an automatic halt is worse than at least giving the people involved a CHANCE to try and get to a state where it's safe to take the system offline and restart it.
- Tomte 4y ago
  You won't see Linux in those fields. I'm aware of a project by OSADL to qualify Linux for SIL 2, but your examples are way beyond that.
  - retard91949 4y ago
    SUSE was used on Mars https://www.pcmag.com/news/linux-is-now-on-mars-thanks-to-nasas-perseverance-rover https://www.pcmag.com/news/linux-is-now-on-mars-thanks-to-na...
- ok_dad 4y ago
  > What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? Hitting an assert in those kinds of systems is a very bad place to be, but an automatic halt is worse than at least giving the people involved a CHANCE to try and get to a state where it's safe to take the system offline and restart it. Kinda a strawman there. That's got to account for, what, 0.0001% of all use of computers, and probably they would never ever use Linux for these applications (I know medical devices DO NOT use Linux).
  - gmueckl 4y ago
    Do you know absolutely every medical device in existence and do you know how broad the definition of a medical device is (including e.g. the monitor attached to the PC used for displaying X-ray images)?
    - jmillikin 4y ago
      > including e.g. the monitor attached to the PC used for displaying > X-ray images Somewhat off-topic, but I used to work in a dental office. The monitors used for displaying X-rays were just normal monitors, bought from Amazon or Newegg or whatever big-box store had a clearance sale. Even the X-ray sensors themselves were (IIRC) not regulated devices, you could buy one right now on AliExpress if you wanted to.
      - gmueckl 4y ago
        That's not the case in the EU. I've worked for an equipment manufacturer for dental clinics. While the monitors were allowed to be off the shelf, the operator (dental clinic) is required to make sure that they work properly and display the image correctly - obey certain brightness and color resolution/calibration standards. Our display software had to refuse to work on an uncalibrated monitor.
        alias_neo 4y ago
        Interesting, how does your software detect an uncalibrated monitor? Did it come with a calibration device which had to be used to scan the display output to check? I don't suppose monitors report calibration data back to display adapters do they?
- notacoward 4y ago
  > What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? The proper answer to those is redundancy, not continuing in an unknown and quite likely harmful state.
  - yencabulator 4y ago
    The leap second bug would have crashed all nodes of a redundant system, at the same time...
    - notacoward 4y ago
      Perhaps. On the other hand, letting a medical device continue moving an actuator or dispensing a medication when it's known to be in a bad "never happen" state could also be fatal. Ditto for the "life support in space" example. Ditto for anything reliant on position, where the system suddenly realizes it has no idea whether its position is correct. Imagine that e.g. on a warship. Limiting responses to external inputs (including time adjustments) can ameliorate such problems. So can software diversity. Many safety-critical systems require one or both, and other measures as well. Picking one black-swan event while ignoring literally every day scenarios doesn't seem very helpful. That's especially true when the thing you're advocating is what actually happened and led to its own bad outcomes.
      - yencabulator 4y ago
        Picking medical devices and warships is also quite the cherry picking. Most Linuxes aren't like that. Critical embedded systems tend to have a hard realtime component, and if Linux is on the system it sits under e.g. seL4, or on a different CPU. At the end of the day, what Linux does is what Linus wants out of it. He's stated, often, that halting the CPU at the exact moment something goes wrong is not the goal. If your goal is to do that, you might not be able to use Linux. If your goal is to put Rust in the Linux kernel, you might have to let go of your goal.
        notacoward 4y ago
- acje 4y ago
  I remember working on some telecom equipment in the 90ies. It had a x86/Unix feature rich distributed management system. In other words complicated and expected to fail. The solution was a “watch dog” circuit that the main CPU had to poll every 100ms or so. Three misses and the CPU would get hard rebooted by the watch dog. This reminds me of two things. Good system design needs a hardware-software codesign. Oxide computers has identified this, but it was probably much more common before the 90ies than after. The second thing is that all things can fail so a strategy that only hardens the one component is fundamentally limited, even flawed. If the component must not fail you need redundancy and supervision. Joe Armstrong would be my source of quote if I needed to find one. Both rust and Linux has some potential for improvement here, but the best answers may lie in their relation to the greater system, rather than within it self. I’m thinking of WASM and hardware codesign respectively.
- stouset 4y ago
  > What about a medical procedure that WILL kill the patient if interrupted? Allow me to introduce you to Therac-25: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- kaba0 4y ago
  That’s why monitoring, fail-safe power offs and redundant systems are important. E.g. even at the complete failure of a CAT scanner’s higher level control (which let’s say would run on an embedded linux kernel), the system would safely stop the radiation and power off, without any instructions from an OS. Here, an inconsistent state from the OS is actually more dangerous than stopping in the middle (e.g. the OS stucks and the same, high energy radiation is continuously being released)
pca006132 4y ago
Probably not very linux, but consider the case where a device is controlling a motor speed by setting its current, computed by getting its current speed and do a simple PID control. If your device crashed (due to failure of something else, for example some LED display) while the current output is pretty large, you may cause some serious damage to whatever that motor is connected to. The thing the system should be able to do is notify such error and gracefully shutdown, for example sending commands to shutdown everything else. Although I think this can be better done by some special panic handler that performs a sjlj and notify other systems about the failure, without continuing running with the wrong output...
- Gare 4y ago
  Such a device should have a simple protective circuit that doesn't allow this. This is common in any expensive or critical industrial system.
  - yellowapple 4y ago
    "Should" unfortunately ain't the same as "does". The Torvaldsian (for lack of a better word) attitude seems to be to assume that someone is indeed dumb enough to design a system wherein all safety measures are software-defined, and in such a situation the software in question probably shouldn't catastrophically fail on every last failed assertion.
MiguelX413 4y ago
What are those sysctls? It was worth my time to read Hacker News this morning.
- stjohnswarts 4y ago
  Maybe these? https://www.cyberciti.biz/tips/reboot-or-halt-linux-system-in-emergency.html https://www.cyberciti.biz/tips/reboot-or-halt-linux-system-i...
UncleMeat 4y ago
It is also a fabulous way of introducing vulnerabilities.
amluto 4y ago
As a kernel developer, I mostly disagree. Panicking hard is nice unless you are the user whose system rebooted without explanation or the developer trying to handle the bug report saying “my system rebooted and I have nothing more to say”. Getting logs out is critical.
- EdSchouten 4y ago
  One does not rule out the other. You could simply write crash info to some NVRAM or something, and then do a reboot. Then you can recover it during the next boot. But there is no need to let userspace processes continue to run, which is exactly what Linux does.
  - wtallis 4y ago
    > You could simply write crash info to some NVRAM or something, and then do a reboot. Then you can recover it during the next boot. That works for some systems: those for which "some NVRAM or something" evaluates to a real device usable for that purpose. Not all Linux systems provide such a device. > But there is no need to let userspace processes continue to run, which is exactly what Linux does. Userspace processes usually contain state that the user would also like to be persisted before rebooting. If my WiFi driver crashes, there's nothing helpful or safer about immediately bringing down the whole system when it's possible to keep running with everything but networking still functioning.
    - EdSchouten 4y ago
      > If my WiFi driver crashes, there's nothing helpful or safer about immediately bringing down the whole system when it's possible to keep running with everything but networking still functioning. There have been various examples of WiFi driver bugs leading to security issues. Didn’t some Broadcom WiFi driver once have a bug in how it processed non-ASCII SSID names, allowing you to trigger remote code execution?
      - wtallis 4y ago
notacoward 4y ago
Yeah, this part has never really been true. > In the kernel, "panic and stop" is not an option That's simply not true. It's an option I've seen exercised many times, even in default configurations. Furthermore, for some domains - e.g. storage - it's the only sane option. Continuing when the world is clearly crazy risks losing or corrupting data, and that's far worse than a crash. No, it's not weird to think all types of computation are ephemeral or less important than preserving the integrity of data. Especially in a distributed context, where this machine might be one of thousands which can cover for a transient loss of one component but letting it continue to run puts everything at risk, rebooting is clearly the better option. A system that can't survive such a reboot is broken. See also: Erlang OTP, Recovery Oriented Computing @ Berkeley. Linus is right overall, but that particular argument is a very bad one. There are systems where "panic and stop" is not an option and there are systems where it's the only option.
- wtallis 4y ago
  > Furthermore, for some domains - e.g. storage - it's the only sane option. Can you elaborate on this? Because failing storage is a common occurrence that usually does not warrant immediately crashing the whole OS, unless it's the root filesystem that becomes inaccessible.
  - notacoward 4y ago
    Depends on what you mean by "failing storage" but IMX it does warrant an immediate stop (with or without reboot depending on circumstances). Yes, for some kinds of media errors it's reasonable to continue, or at least not panic. Another option in some cases is to go read-only. OTOH, if either media or memory corruption is detected, it would almost certainly be unsafe to continue because that might lead to writing the wrong data or writing it to the wrong place. The general rule in storage is that inaccessible data is preferable to lost, corrupted, or improperly overwritten data. Especially in a distributed storage system using erasure codes etc., losing one machine means absolutely nothing even if it's permanent. On the last storage project I worked on, we routinely ran with 1-5% of machines down, whether it was due to failures or various kinds of maintenance actions, and all it meant was a loss of some capacity/performance. It's what the system was designed for. Leaving a faulty machine running, OTOH, could have led to a Byzantine failure mode corrupting all shards for a block and thus losing its contents forever. BTW, in that sort of context - where most bytes in the world are held BTW - the root filesystem is more expendable than any other. It's just part of the access system, much like firmware, and re-imaging or even hardware replacement doesn't affect the real persistence layer. It's user data that must be king, and those media whose contents must be treated with the utmost care.

xyzzy_plugh 4y ago

Right, the fact that you can toggle those sysctls is sort of the point. There are plenty of environments where running at all is better than not, hardware watchdogs can solve for unresponsiveness. TFA is about making it possible for the kernel to decide what to do, rather than exploding on the spot, which is terrible.

deleted 4y ago

[deleted]

mike_hock 4y ago

So instead of a power spike, we'd have had a major internet outage across the world, across the entire industry and beyond, probably, if everyone had panicked on oops. The blame really lies with people not monitoring their systems. As you said, you have the option to reboot on panic, but Linus is absolutely not wrong that this size does not fit all. What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? Hitting an assert in those kinds of systems is a very bad place to be, but an automatic halt is worse than at least giving the people involved a CHANCE to try and get to a state where it's safe to take the system offline and restart it.

Tomte 4y ago
You won't see Linux in those fields. I'm aware of a project by OSADL to qualify Linux for SIL 2, but your examples are way beyond that.
- retard91949 4y ago
  SUSE was used on Mars https://www.pcmag.com/news/linux-is-now-on-mars-thanks-to-nasas-perseverance-rover https://www.pcmag.com/news/linux-is-now-on-mars-thanks-to-na...
ok_dad 4y ago
> What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? Hitting an assert in those kinds of systems is a very bad place to be, but an automatic halt is worse than at least giving the people involved a CHANCE to try and get to a state where it's safe to take the system offline and restart it. Kinda a strawman there. That's got to account for, what, 0.0001% of all use of computers, and probably they would never ever use Linux for these applications (I know medical devices DO NOT use Linux).
- gmueckl 4y ago
  Do you know absolutely every medical device in existence and do you know how broad the definition of a medical device is (including e.g. the monitor attached to the PC used for displaying X-ray images)?
  - jmillikin 4y ago
    > including e.g. the monitor attached to the PC used for displaying > X-ray images Somewhat off-topic, but I used to work in a dental office. The monitors used for displaying X-rays were just normal monitors, bought from Amazon or Newegg or whatever big-box store had a clearance sale. Even the X-ray sensors themselves were (IIRC) not regulated devices, you could buy one right now on AliExpress if you wanted to.
    - gmueckl 4y ago
      That's not the case in the EU. I've worked for an equipment manufacturer for dental clinics. While the monitors were allowed to be off the shelf, the operator (dental clinic) is required to make sure that they work properly and display the image correctly - obey certain brightness and color resolution/calibration standards. Our display software had to refuse to work on an uncalibrated monitor.
      - alias_neo 4y ago
        Interesting, how does your software detect an uncalibrated monitor? Did it come with a calibration device which had to be used to scan the display output to check? I don't suppose monitors report calibration data back to display adapters do they?
notacoward 4y ago
> What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? The proper answer to those is redundancy, not continuing in an unknown and quite likely harmful state.
- yencabulator 4y ago
  The leap second bug would have crashed all nodes of a redundant system, at the same time...
  - notacoward 4y ago
    Perhaps. On the other hand, letting a medical device continue moving an actuator or dispensing a medication when it's known to be in a bad "never happen" state could also be fatal. Ditto for the "life support in space" example. Ditto for anything reliant on position, where the system suddenly realizes it has no idea whether its position is correct. Imagine that e.g. on a warship. Limiting responses to external inputs (including time adjustments) can ameliorate such problems. So can software diversity. Many safety-critical systems require one or both, and other measures as well. Picking one black-swan event while ignoring literally every day scenarios doesn't seem very helpful. That's especially true when the thing you're advocating is what actually happened and led to its own bad outcomes.
    - yencabulator 4y ago
      Picking medical devices and warships is also quite the cherry picking. Most Linuxes aren't like that. Critical embedded systems tend to have a hard realtime component, and if Linux is on the system it sits under e.g. seL4, or on a different CPU. At the end of the day, what Linux does is what Linus wants out of it. He's stated, often, that halting the CPU at the exact moment something goes wrong is not the goal. If your goal is to do that, you might not be able to use Linux. If your goal is to put Rust in the Linux kernel, you might have to let go of your goal.
      - notacoward 4y ago
acje 4y ago
I remember working on some telecom equipment in the 90ies. It had a x86/Unix feature rich distributed management system. In other words complicated and expected to fail. The solution was a “watch dog” circuit that the main CPU had to poll every 100ms or so. Three misses and the CPU would get hard rebooted by the watch dog. This reminds me of two things. Good system design needs a hardware-software codesign. Oxide computers has identified this, but it was probably much more common before the 90ies than after. The second thing is that all things can fail so a strategy that only hardens the one component is fundamentally limited, even flawed. If the component must not fail you need redundancy and supervision. Joe Armstrong would be my source of quote if I needed to find one. Both rust and Linux has some potential for improvement here, but the best answers may lie in their relation to the greater system, rather than within it self. I’m thinking of WASM and hardware codesign respectively.
stouset 4y ago
> What about a medical procedure that WILL kill the patient if interrupted? Allow me to introduce you to Therac-25: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
kaba0 4y ago
That’s why monitoring, fail-safe power offs and redundant systems are important. E.g. even at the complete failure of a CAT scanner’s higher level control (which let’s say would run on an embedded linux kernel), the system would safely stop the radiation and power off, without any instructions from an OS. Here, an inconsistent state from the OS is actually more dangerous than stopping in the middle (e.g. the OS stucks and the same, high energy radiation is continuously being released)

pca006132 4y ago

Probably not very linux, but consider the case where a device is controlling a motor speed by setting its current, computed by getting its current speed and do a simple PID control. If your device crashed (due to failure of something else, for example some LED display) while the current output is pretty large, you may cause some serious damage to whatever that motor is connected to. The thing the system should be able to do is notify such error and gracefully shutdown, for example sending commands to shutdown everything else. Although I think this can be better done by some special panic handler that performs a sjlj and notify other systems about the failure, without continuing running with the wrong output...

Gare 4y ago
Such a device should have a simple protective circuit that doesn't allow this. This is common in any expensive or critical industrial system.
- yellowapple 4y ago
  "Should" unfortunately ain't the same as "does". The Torvaldsian (for lack of a better word) attitude seems to be to assume that someone is indeed dumb enough to design a system wherein all safety measures are software-defined, and in such a situation the software in question probably shouldn't catastrophically fail on every last failed assertion.

MiguelX413 4y ago

What are those sysctls? It was worth my time to read Hacker News this morning.

stjohnswarts 4y ago
Maybe these? https://www.cyberciti.biz/tips/reboot-or-halt-linux-system-in-emergency.html https://www.cyberciti.biz/tips/reboot-or-halt-linux-system-i...

UncleMeat 4y ago

It is also a fabulous way of introducing vulnerabilities.

amluto 4y ago

As a kernel developer, I mostly disagree. Panicking hard is nice unless you are the user whose system rebooted without explanation or the developer trying to handle the bug report saying “my system rebooted and I have nothing more to say”. Getting logs out is critical.

EdSchouten 4y ago
One does not rule out the other. You could simply write crash info to some NVRAM or something, and then do a reboot. Then you can recover it during the next boot. But there is no need to let userspace processes continue to run, which is exactly what Linux does.
- wtallis 4y ago
  > You could simply write crash info to some NVRAM or something, and then do a reboot. Then you can recover it during the next boot. That works for some systems: those for which "some NVRAM or something" evaluates to a real device usable for that purpose. Not all Linux systems provide such a device. > But there is no need to let userspace processes continue to run, which is exactly what Linux does. Userspace processes usually contain state that the user would also like to be persisted before rebooting. If my WiFi driver crashes, there's nothing helpful or safer about immediately bringing down the whole system when it's possible to keep running with everything but networking still functioning.
  - EdSchouten 4y ago
    > If my WiFi driver crashes, there's nothing helpful or safer about immediately bringing down the whole system when it's possible to keep running with everything but networking still functioning. There have been various examples of WiFi driver bugs leading to security issues. Didn’t some Broadcom WiFi driver once have a bug in how it processed non-ASCII SSID names, allowing you to trigger remote code execution?
    - wtallis 4y ago

notacoward 4y ago

Yeah, this part has never really been true. > In the kernel, "panic and stop" is not an option That's simply not true. It's an option I've seen exercised many times, even in default configurations. Furthermore, for some domains - e.g. storage - it's the only sane option. Continuing when the world is clearly crazy risks losing or corrupting data, and that's far worse than a crash. No, it's not weird to think all types of computation are ephemeral or less important than preserving the integrity of data. Especially in a distributed context, where this machine might be one of thousands which can cover for a transient loss of one component but letting it continue to run puts everything at risk, rebooting is clearly the better option. A system that can't survive such a reboot is broken. See also: Erlang OTP, Recovery Oriented Computing @ Berkeley. Linus is right overall, but that particular argument is a very bad one. There are systems where "panic and stop" is not an option and there are systems where it's the only option.

wtallis 4y ago
> Furthermore, for some domains - e.g. storage - it's the only sane option. Can you elaborate on this? Because failing storage is a common occurrence that usually does not warrant immediately crashing the whole OS, unless it's the root filesystem that becomes inaccessible.
- notacoward 4y ago
  Depends on what you mean by "failing storage" but IMX it does warrant an immediate stop (with or without reboot depending on circumstances). Yes, for some kinds of media errors it's reasonable to continue, or at least not panic. Another option in some cases is to go read-only. OTOH, if either media or memory corruption is detected, it would almost certainly be unsafe to continue because that might lead to writing the wrong data or writing it to the wrong place. The general rule in storage is that inaccessible data is preferable to lost, corrupted, or improperly overwritten data. Especially in a distributed storage system using erasure codes etc., losing one machine means absolutely nothing even if it's permanent. On the last storage project I worked on, we routinely ran with 1-5% of machines down, whether it was due to failures or various kinds of maintenance actions, and all it meant was a loss of some capacity/performance. It's what the system was designed for. Leaving a faulty machine running, OTOH, could have led to a Byzantine failure mode corrupting all shards for a block and thus losing its contents forever. BTW, in that sort of context - where most bytes in the world are held BTW - the root filesystem is more expendable than any other. It's just part of the access system, much like firmware, and re-imaging or even hardware replacement doesn't affect the real persistence layer. It's user data that must be king, and those media whose contents must be treated with the utmost care.