58 ms·

>[The fact that computers are made of many components separated by communication buses] suggests that it may be possible to abstract away the distributed nature

>[The fact that computers are made of many components separated by communication buses] suggests that it may be possible to abstract away the distributed nature of larger-scale systems. This is a neat line of thought, but I don't think it can go very far. There is a huge difference in reliability and predictability between small-scale and large-scale systems. One way to see this is to look at power supplies. Two ICs on the same board can be running off of the same 3.3V supply, and will almost certainly have a single upstream AC connection to the mains. When thinking about communications between the ICs, you don't have to consider power failure because a power failure will take down both ICs. Compare this to a WiFi network where two devices could be on separate parts of the power grid! Other kinds of failures are rare enough to be ignored completely for most applications. An Ethernet cable can be unplugged. A PCB trace can't. I used to work with a low-level digital communication protocol called I²C. It's designed for communication between two chips on the same board. There is no defined timeout for communication. A single malfunctioning slave device can hang the entire bus. According to the official protocol spec, the recommended way of dealing with this is to reset every device on the bus (which may mean resetting the entire board). If a hardware reset is not available, the recommendation is to power-cycle the system! [1] Now I²C is a particularly sloppy protocol, and higher-level versions (SMBus and PMBus) do fix these problems, so this is a bit of an extreme example. But the fact that I²C is still commonly used today shows how reliable a small-scale electronic system can be. Even at the PC level, low-level hardware faults are rare enough that they're often indicated only by weird behavior ("My system hangs when the GPU gets hot"), and the solution is often for the user to guess which component is broken and replace it. [1] Section 3.1.16 of https://www.nxp.com/docs/en/user-guide/UM10204.pdf https://www.nxp.com/docs/en/user-guide/UM10204.pdf

charcircuit 5y ago
>A PCB trace can't. Sure, but physical damage can disconnect it.
- anyfoo 5y ago
  But in that case we accept the device as "broken" and need to replace or repair it. If you are lucky the PCB trace was only relevant to a single feature, say an LED. If you are unlucky, it's part of the memory bus and everything is toast. But no significant engineering went into making one more resilient than the other. Whereas in a distributed system, a single broken communication line, even or especially if it's extremely important, still means that the distributed system has to recover somehow, preferably gracefully.
  - charcircuit 5y ago
    Making things robust to failure is an engineering trade off. Your comment simply points out that there are cases where handling failures aren't seen as being important.
    - harry8 5y ago
      Clearly it's a trade off as everything always is. So you deal with probabilities of various kinds of failure in various circumstances. Turning on a computer, compare the probability of an issue with the ethernet cable, ie not plugged in as suggested vs a pcb trace being broken. Given that probability do you want to handle one of those gracefully and not fry the machine? Both? Neither. And that's the point of identifying the difference. A thing that is rare and if it occurs everything is b0rked anyway doesn't need to be handled. That individual components having to be robust on some level (what level? It depends...) to failure difference of distribution via a network connection and pcb traces is probably the reason we don't usually (but we might) refer to the latter as a distributed system.
littleJeck 5y ago
> Now I²C is a particularly sloppy protocol I know this pain as an embedded software monkey. So many edge cases unthought of and no method to gracefully fail. Not to mention you always find some fault with the I²C implementation on the SOC or slave device you want to talk to. I’ve spent weeks debugging bus lockup problems only to find erratas for the silicon, or find this particular chip needs an extra 100ms to startup than others. It’s the one communication protocol that o always have problems with. Nothing else has caused me as many headaches.
- a9h74j 5y ago
  This is something which worries me with Oxide using I2C (according to their board bring-up podcast), and using an auto-direction-switching level-shifter on the I2C bus. I hope and I am sure they can make it work, but to me that comes closer to violating a (core?) value of determinism and observability. Of course if it is only one well-studied component or two, perhaps out of design/supply necessity, this might be a very small risk in the larger scope.
  - AdamH12113 5y ago
    Are you sure it’s really I2C? Servers often use PMBus for power supply management, which uses the I2C physical layer but the (better-behaved) SMBus link layer. I’m not familiar with Oxide, but the I2C physical layer is just a pair of open-drain signals, so I wouldn’t worry about a level-shifter too much.
  - bcantrill 5y ago
    It is indeed a very small risk in the larger scope -- and besides, using I2C isn't optional: all of the devices on the board (from converters/regulators, temp sensors, clock generators and fan controllers to CPUs, DIMMs, NICs, and NVMe drives) have I2C (or I2C-based) management interfaces. If you'd like more concrete detail, take a look at the Hubris app definition for our Gimlet board.[0] [0] https://github.com/oxidecomputer/hubris/blob/master/app/gimlet/rev-b.toml https://github.com/oxidecomputer/hubris/blob/master/app/giml...
    - 5y ago

charcircuit 5y ago

>A PCB trace can't. Sure, but physical damage can disconnect it.

anyfoo 5y ago
But in that case we accept the device as "broken" and need to replace or repair it. If you are lucky the PCB trace was only relevant to a single feature, say an LED. If you are unlucky, it's part of the memory bus and everything is toast. But no significant engineering went into making one more resilient than the other. Whereas in a distributed system, a single broken communication line, even or especially if it's extremely important, still means that the distributed system has to recover somehow, preferably gracefully.
- charcircuit 5y ago
  Making things robust to failure is an engineering trade off. Your comment simply points out that there are cases where handling failures aren't seen as being important.
  - harry8 5y ago
    Clearly it's a trade off as everything always is. So you deal with probabilities of various kinds of failure in various circumstances. Turning on a computer, compare the probability of an issue with the ethernet cable, ie not plugged in as suggested vs a pcb trace being broken. Given that probability do you want to handle one of those gracefully and not fry the machine? Both? Neither. And that's the point of identifying the difference. A thing that is rare and if it occurs everything is b0rked anyway doesn't need to be handled. That individual components having to be robust on some level (what level? It depends...) to failure difference of distribution via a network connection and pcb traces is probably the reason we don't usually (but we might) refer to the latter as a distributed system.

littleJeck 5y ago

> Now I²C is a particularly sloppy protocol I know this pain as an embedded software monkey. So many edge cases unthought of and no method to gracefully fail. Not to mention you always find some fault with the I²C implementation on the SOC or slave device you want to talk to. I’ve spent weeks debugging bus lockup problems only to find erratas for the silicon, or find this particular chip needs an extra 100ms to startup than others. It’s the one communication protocol that o always have problems with. Nothing else has caused me as many headaches.

a9h74j 5y ago
This is something which worries me with Oxide using I2C (according to their board bring-up podcast), and using an auto-direction-switching level-shifter on the I2C bus. I hope and I am sure they can make it work, but to me that comes closer to violating a (core?) value of determinism and observability. Of course if it is only one well-studied component or two, perhaps out of design/supply necessity, this might be a very small risk in the larger scope.
- AdamH12113 5y ago
  Are you sure it’s really I2C? Servers often use PMBus for power supply management, which uses the I2C physical layer but the (better-behaved) SMBus link layer. I’m not familiar with Oxide, but the I2C physical layer is just a pair of open-drain signals, so I wouldn’t worry about a level-shifter too much.
- bcantrill 5y ago
  It is indeed a very small risk in the larger scope -- and besides, using I2C isn't optional: all of the devices on the board (from converters/regulators, temp sensors, clock generators and fan controllers to CPUs, DIMMs, NICs, and NVMe drives) have I2C (or I2C-based) management interfaces. If you'd like more concrete detail, take a look at the Hubris app definition for our Gimlet board.[0] [0] https://github.com/oxidecomputer/hubris/blob/master/app/gimlet/rev-b.toml https://github.com/oxidecomputer/hubris/blob/master/app/giml...
  - 5y ago