5 ms·

Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing. Sure, storage devices can come and go, but that's been the case

by hsn915 5y ago

Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing. Sure, storage devices can come and go, but that's been the case since forever, and most programs are not written to handle this edge case gracefully. Except for the OS kernel. The links between the components of your computer are solid and cannot fail like actual computer network connections. In terms of "CAP" theorom, the system has no Partition tolerance. If one of the the links connecting CPUs/GPUs/RAM breaks, all hell breaks loose. If a single instruction is not processed correctly, all hell might break loose. So I find the analogy misleading.

aidenn0 5y ago
I think that TFA gets it exactly backwards. It's not that we will be able to treat multi-node systems as non-distributed it's that single-nodes will have to start being treated like distributed systems. > The links between the components of your computer are solid and cannot fail like actual computer network connections. I've personally had this disproven to me on multiple occasions.
- catern 5y ago
  >I've personally had this disproven to me on multiple occasions. That sounds like interesting stories! Can you elaborate?
  - aidenn0 5y ago
    Accidents on desktop hardware: Multiple bad disk cables (more common in IDE era, but happened once with SATA). Interestingly enough, Windows would reduce the drive speed on certain errors, so I had a drive that booted up in UDMA/133 and the longer it was running the slower it got, eventually settling in at PIO mode 2. Switching the drive cable fixed it. A sound card that wasn't screwed in to the case, so if you pushed the phone connector in too hard it would unseat. I still don't know how that happened; it must have been me (unless someone pranked me) but the sound-card hadn't been changed in like 2 years at that point. A DIMM wasn't fully clipped in, but the system worked fine for weeks until someone bumped into the case. Things that were actually intentional: We expect anything plugged in externally (e.g. USB, ethernet, HDMI) to be plugged and unplugged without needing to restart the system. This sounds banal, but wasn't always the case. I had a network card with 3 interfaces (10BASE5 AUI, 10BASE2 BNC, 10BASE-T modular plug) and you needed to power off the system and toggle a DIP switch to change which was in use. I've seen server and minicomputer hardware with hotpluggable CPUs and RAM Eurocard type systems (e.g. VME, cPCI) could connect all sorts of things, and could run without restarting. This sort of blurs the line as to what a "node" is. If you have multiple CPUs on the same PCI bus, is that one node or many? eGPUs have made hotplugging a GPU something that anyone might do today. If you run this setup, then the majority of the computational power in your system can appear and disappear at will, along with multiple GB of RAM.
StillBored 5y ago
There have been machines tolerant to CPU and Mem failures, and to a certain extent this sorta works on some of the higher end machines that support ram/cpu hotplug. (historically see hp/tandem/nonstop, sunos/imp, etc). The problem is linux's monolithic model, doesn't work well for kernel checkpoint/restore despite it actually supporting hotplug cpu/ram it they have to be gracefully removed. So, this is less about the machine being distributed, and more about the fact that linux is the opposite of a microkernel/etc that can isolate and restart its subsystems in the face of failure. Its also sorta funny that while these types of operations tend to need to be designed into the system, the last major OS's designed this way were done in the 1980's.
- dwohnitmok 5y ago
  I know of no OSes that are resilient to CPU cores producing wrong results (or incorrect mem results: I consider ECC a lower level concern that is not part of the OS), whereas a lot of distributed consensus algorithms have this built into their requirements. EDIT: I have heard through the grapevine that something like this might be done for aerospace, but I have no personal experience with that. I agree with parent. The major reason why programming on a single computer is easier than a distributed system is that we assume total resilience of various components that we cannot for a distributed system. From the article: > This offers hope that it is possible to some day abstract away the distributed nature of larger-scale systems. To do this is not a question of software abstractions, but hardware resilience. If we have a network which we can reasonably assume to have 100% uptime and absolutely no corruption between all its components then we can program distributed systems as single computers.
  - catern 5y ago
    Most distributed consensus algorithms, or distributed systems in general, are not resilient to nodes producing arbitrary wrong results. That's the realm of systems like Bitcoin, which achieve such resilience by paying big performance costs. So it shouldn't be surprising that computers have the same lack of resilience.
    - 5y ago
catern 5y ago
>Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing That's incorrect. There are plenty of machines/OSs which are (or can be) resilient to a CPU failing; Linux, for example. From the OS point of view, you just kill the process that was running on the CPU at the time and move on. Resilience to spontaneous RAM failures is rarer but possible.
- bee_rider 5y ago
  Killing the processes running on the compute element seems not very graceful, right? I'd expect a gracefully handled failure to have some state staved from which the computation can be continued. Which would be overkill on a single node, given that CPUs don't really fail all that often.
  - catern 5y ago
    It's up to userspace to do more than that. There are other issues which can cause processes to be spontaneously killed (OOMkiller for example) so it's something you should be tolerant of.
    - nickelpro 5y ago
      Disagree. An environment that's being reaped by OOMK is not stable enough to make assumptions about. You're in "go down the hall and turn it off and on again" territory. Attempting to account for such environments in user programs massively inflates their complexity, does little to enhance reliability, and the resulting behavior is typically brittle or outright broken from the get go. This is why, for example, the C++ committee flirts with making allocation failure a UB condition.
      - astrange 5y ago
        iOS constantly runs an OOM killer and works pretty well.
imtringued 5y ago
The article also ignores that e.g. the CUDA API looks nothing like a local function call. People are explicitly aware when they are launching GPU kernels.
bee_rider 5y ago
You can 'disable' a core in Linux pretty easily, although I'm not sure to what extent you'd consider this graceful (in the sense that you write to a system file and then some magic, which may be arbitrarily complicated I guess, happens in the background. So it doesn't seem equivalent to just yanking a core from the package, if that were possible).

by hsn915 5y ago

aidenn0 5y ago

I think that TFA gets it exactly backwards. It's not that we will be able to treat multi-node systems as non-distributed it's that single-nodes will have to start being treated like distributed systems. > The links between the components of your computer are solid and cannot fail like actual computer network connections. I've personally had this disproven to me on multiple occasions.

catern 5y ago
>I've personally had this disproven to me on multiple occasions. That sounds like interesting stories! Can you elaborate?
- aidenn0 5y ago
  Accidents on desktop hardware: Multiple bad disk cables (more common in IDE era, but happened once with SATA). Interestingly enough, Windows would reduce the drive speed on certain errors, so I had a drive that booted up in UDMA/133 and the longer it was running the slower it got, eventually settling in at PIO mode 2. Switching the drive cable fixed it. A sound card that wasn't screwed in to the case, so if you pushed the phone connector in too hard it would unseat. I still don't know how that happened; it must have been me (unless someone pranked me) but the sound-card hadn't been changed in like 2 years at that point. A DIMM wasn't fully clipped in, but the system worked fine for weeks until someone bumped into the case. Things that were actually intentional: We expect anything plugged in externally (e.g. USB, ethernet, HDMI) to be plugged and unplugged without needing to restart the system. This sounds banal, but wasn't always the case. I had a network card with 3 interfaces (10BASE5 AUI, 10BASE2 BNC, 10BASE-T modular plug) and you needed to power off the system and toggle a DIP switch to change which was in use. I've seen server and minicomputer hardware with hotpluggable CPUs and RAM Eurocard type systems (e.g. VME, cPCI) could connect all sorts of things, and could run without restarting. This sort of blurs the line as to what a "node" is. If you have multiple CPUs on the same PCI bus, is that one node or many? eGPUs have made hotplugging a GPU something that anyone might do today. If you run this setup, then the majority of the computational power in your system can appear and disappear at will, along with multiple GB of RAM.

StillBored 5y ago

There have been machines tolerant to CPU and Mem failures, and to a certain extent this sorta works on some of the higher end machines that support ram/cpu hotplug. (historically see hp/tandem/nonstop, sunos/imp, etc). The problem is linux's monolithic model, doesn't work well for kernel checkpoint/restore despite it actually supporting hotplug cpu/ram it they have to be gracefully removed. So, this is less about the machine being distributed, and more about the fact that linux is the opposite of a microkernel/etc that can isolate and restart its subsystems in the face of failure. Its also sorta funny that while these types of operations tend to need to be designed into the system, the last major OS's designed this way were done in the 1980's.

dwohnitmok 5y ago
I know of no OSes that are resilient to CPU cores producing wrong results (or incorrect mem results: I consider ECC a lower level concern that is not part of the OS), whereas a lot of distributed consensus algorithms have this built into their requirements. EDIT: I have heard through the grapevine that something like this might be done for aerospace, but I have no personal experience with that. I agree with parent. The major reason why programming on a single computer is easier than a distributed system is that we assume total resilience of various components that we cannot for a distributed system. From the article: > This offers hope that it is possible to some day abstract away the distributed nature of larger-scale systems. To do this is not a question of software abstractions, but hardware resilience. If we have a network which we can reasonably assume to have 100% uptime and absolutely no corruption between all its components then we can program distributed systems as single computers.
- catern 5y ago
  Most distributed consensus algorithms, or distributed systems in general, are not resilient to nodes producing arbitrary wrong results. That's the realm of systems like Bitcoin, which achieve such resilience by paying big performance costs. So it shouldn't be surprising that computers have the same lack of resilience.
  - 5y ago

catern 5y ago

>Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing That's incorrect. There are plenty of machines/OSs which are (or can be) resilient to a CPU failing; Linux, for example. From the OS point of view, you just kill the process that was running on the CPU at the time and move on. Resilience to spontaneous RAM failures is rarer but possible.

bee_rider 5y ago
Killing the processes running on the compute element seems not very graceful, right? I'd expect a gracefully handled failure to have some state staved from which the computation can be continued. Which would be overkill on a single node, given that CPUs don't really fail all that often.
- catern 5y ago
  It's up to userspace to do more than that. There are other issues which can cause processes to be spontaneously killed (OOMkiller for example) so it's something you should be tolerant of.
  - nickelpro 5y ago
    Disagree. An environment that's being reaped by OOMK is not stable enough to make assumptions about. You're in "go down the hall and turn it off and on again" territory. Attempting to account for such environments in user programs massively inflates their complexity, does little to enhance reliability, and the resulting behavior is typically brittle or outright broken from the get go. This is why, for example, the C++ committee flirts with making allocation failure a UB condition.
    - astrange 5y ago
      iOS constantly runs an OOM killer and works pretty well.

imtringued 5y ago

The article also ignores that e.g. the CUDA API looks nothing like a local function call. People are explicitly aware when they are launching GPU kernels.

bee_rider 5y ago

You can 'disable' a core in Linux pretty easily, although I'm not sure to what extent you'd consider this graceful (in the sense that you write to a system file and then some magic, which may be arbitrarily complicated I guess, happens in the background. So it doesn't seem equivalent to just yanking a core from the package, if that were possible).