4 ms·

The author has clearly never chased a heisenbug in a 25 year old Makefile for embedded systems. The kind where a timing bug, plus NFS automounter, plus WAN weat

The author has clearly never chased a heisenbug in a 25 year old Makefile for embedded systems. The kind where a timing bug, plus NFS automounter, plus WAN weather, equals an unexpected PATH, resulting in a different minor version of a linker than expected, thereby changing the order of static initializers on some targets, thereby resulting in sshd not checking passwords, but only on MIPS and only with the 16 mbps flash config.

fit2rule 7y ago
.. you thus provide a perfect example of the claim made.
evmar 7y ago
The author had a followup post about this that I found more interesting: https://blog.nelhage.com/post/systems-that-defy-understanding/ https://blog.nelhage.com/post/systems-that-defy-understandin...
- munchbunny 7y ago
  As someone who works on a system that spans 10^big nodes, I really like the point in that post. At a certain level of complexity even your most baseline assumptions will have counterexamples, and 99.9% uptime means 1 in 1000 nodes are in various states of failure at any time. You have to build fault tolerance into the design at each layer to keep errors at base layers from snowballing into real outages. And that’s inherently at odds with the “fail fast” principle, so you have to spend a lot of energy on how to fail fast inwardly but stay operational outwardly.
geofft 7y ago
All of those can be understood, no? (I'd expect to go at least that deep in a postmortem of that problem, and I regularly do go about that deep in similar postmortems.)
- ken 7y ago
  That might be stretching the definition of "deterministic". If it depends on implicit state, precise timing, external conditions, etc., I would say it's closer to fitting the definition of "chaotic" than "deterministic". With perfect observation and recording, it may be possible to see a tornado and work backwards to figure out which butterflies flapped their wings to cause it, but that would still be a long ways off from being able to train all butterflies to move only in ways which won't cause tornados.
  - geofft 7y ago
    Sure, but you don't need to know which butterflies flapped their wings - you just need to know "if there's a tornado, the house falls down, and we're in tornado country." In the example above, it seems sufficient and entirely testable to conclude, if for whatever reason NFS is flaky, $PATH resolution will skip the directory with the right linker and fall back to the old one, and if for whatever reason you call the old linker, it has different behavior about static initializer order. That's enough to both describe the problem, reproduce it, and fix it. No, you're not going to be able to identify which particular packet was lost, but (and I say this as someone who's maintained production systems that rely on cross-continent NFS) you expect that packets can be lost and you figure out how to make the system robust against it. Don't put multiple files with the same name on $PATH, or call the linker by its full path, or something. And that leaves you a very far distance from "something spooky is happening with the computer and so passwords weren't being checked today, who knows."

fit2rule 7y ago

.. you thus provide a perfect example of the claim made.

evmar 7y ago

The author had a followup post about this that I found more interesting: https://blog.nelhage.com/post/systems-that-defy-understanding/ https://blog.nelhage.com/post/systems-that-defy-understandin...

munchbunny 7y ago
As someone who works on a system that spans 10^big nodes, I really like the point in that post. At a certain level of complexity even your most baseline assumptions will have counterexamples, and 99.9% uptime means 1 in 1000 nodes are in various states of failure at any time. You have to build fault tolerance into the design at each layer to keep errors at base layers from snowballing into real outages. And that’s inherently at odds with the “fail fast” principle, so you have to spend a lot of energy on how to fail fast inwardly but stay operational outwardly.

geofft 7y ago

All of those can be understood, no? (I'd expect to go at least that deep in a postmortem of that problem, and I regularly do go about that deep in similar postmortems.)

ken 7y ago
That might be stretching the definition of "deterministic". If it depends on implicit state, precise timing, external conditions, etc., I would say it's closer to fitting the definition of "chaotic" than "deterministic". With perfect observation and recording, it may be possible to see a tornado and work backwards to figure out which butterflies flapped their wings to cause it, but that would still be a long ways off from being able to train all butterflies to move only in ways which won't cause tornados.
- geofft 7y ago
  Sure, but you don't need to know which butterflies flapped their wings - you just need to know "if there's a tornado, the house falls down, and we're in tornado country." In the example above, it seems sufficient and entirely testable to conclude, if for whatever reason NFS is flaky, $PATH resolution will skip the directory with the right linker and fall back to the old one, and if for whatever reason you call the old linker, it has different behavior about static initializer order. That's enough to both describe the problem, reproduce it, and fix it. No, you're not going to be able to identify which particular packet was lost, but (and I say this as someone who's maintained production systems that rely on cross-continent NFS) you expect that packets can be lost and you figure out how to make the system robust against it. Don't put multiple files with the same name on $PATH, or call the linker by its full path, or something. And that leaves you a very far distance from "something spooky is happening with the computer and so passwords weren't being checked today, who knows."