16 ms·
On rebooting: the unreasonable effectiveness of turning computers off and on
- xthrowawayxx 4y agoI once deleted a 15TB database by temporarily shutting off an AWS i3en instance. Nobody told i3en's have "ephemeral storage" that gets wiped on shutdown.
- inopinatus 4y agoI know of an AWS customer that achieved a similar outcome for their entire stack after the ops team was directed by finance to use only spot instances.
- jcims 4y agoI imagine a lot of folks have lost toes to that foot gun.
- meerita 4y agoBack in the 90's i competed with my friends on who had the longest uptime. Most guys I've meet they reseted the computer multiple times a day to increase the performance. I never turn off my mac, it has to be a software update or something like a general crash.
- mikenakis 4y agoIs there a citable source for "The parable of Mike (Michael Burrows) and the login shell"? I do not think that this whole discussion about returning to a "known good initial state" has much merit, because computers are Turing machines: they modify their own state and then act upon that state. Which means that unless your boot drive is read-only, nobody can predict what your computer will do on the next reboot. (Unless someone solved The Halting Problem and I missed it.) But the discussion about assertions and initially fragile software that quickly becomes robust did strike a chord with me, because that's exactly what I have been doing for decades now. (Also, I see other programmers using assertions so rarely, that I feel kind of lonely in this.) I have a long post about assertions (https://blog.michael.gr/2014/09/assertions-and-testing.html https://blog.michael.gr/2014/09/assertions-and-testing.html) to which I would like to add a paragraph or two about this aspect, and preferably cite Burrows.
- mjevans 5y agoThe missing link is RE-initilization or re-validation of the expected state. Offhand I remember some discussion of how old dialup friendly multiplayer games would transfer state. Differential state would be transferred. There might or might not be a checksum. There would be global state refreshes (either periodically or as bandwidth allowed). The global state refreshes are a different form of re-initilization. The current state discarded in favor of a canonically approved state.
- foobiekr 5y agoWe call this station keeping or anti entropy in our designs.
- tingletech 5y agoThe first law of tech support. Second law is check (unplug and re-plug) all the cables.
- layer8 5y agoThe winning strategy of Mike’s parable is also known as offensive programming: https://en.m.wikipedia.org/wiki/Offensive_programming https://en.m.wikipedia.org/wiki/Offensive_programming
- RajT88 5y ago> Is this the best that anyone can do? No. It's the best that can sometimes be done quickly. Additionally, this doesn't mention the value of a good postmortem. Or the horrors of cloud computing, where restarting things is deemed a good enough mitigation because in the cloud these things happen, and nobody pushes for a good postmortem and repair items.
- mannykannot 5y agoThe whole article here is an argument for the proposition that, in many cases, this is not just the best compromise taking expediency into account, but the best thing to do, period (there are some caveats in the section titled "Complications.") You are entitled to a contrary view, but you have not said anything to counter the arguments presented here. The last two sections do include the value of postmortems (or forensic analysis, as the author puts it) though from the perspective that this is more feasible and effective on a system that promptly crashes when things go wrong.
- throwawaymaths 4y agoIt sometimes really is as good as you can expect. You can have a compiler that checks for increasingly complex errors in intent but it won't defend you against cosmic rays.
- sbf501 5y agoOpposite of the robustness principle: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle It's funny, I spent some time developing tools for CPU architects. Both the concept in the OP's anecdote, and the above principle don't really apply because logic doesn't break in the same way source code breaks. You don't run a program in HDL, you synthesize a logic flow. One could concievably test all possible combinations of that logic for errors, but it becomes 2^N combinations where N is the number of inputs and the number of state elements. Since this cannot be tested because the space is huge (excluding hierarchical designs and emulation), you generate targeted test patterns (and many many mutexes) to pare down the space, and perhaps randomize some of the targeted tests to verify you don't execute out of bounds. And even "out of bounds" is defined by however smart the microarchitect was when they wrote the spec, and that can be wrong too. The only way to find and fix these bugs is to run trillions of test vectors (called "coverage") and hope you've passed all the known bugs, and not found any new ones. There are four decades of papers written on hardware validation, so I'm barely scratching the surface, but I think it's a very different perspective compared to how programmers approach the world. I think a lot of the bugs that OP is talking about fall into the hardware logic domain. There isn't really a fallback "throw" or "return status" that you can even check for. Just fault handlers (for the most part).
- not2b 4y agoThe hardware world was doing fuzzing about a decade before the software world was, though they called it coverage-directed constrained random simulation: generate test vectors in a way that you cover more of the design.
- sbf501 4y ago
- legalcorrection 4y ago>One could concievably test all possible combinations of that logic for errors, but it becomes 2^N combinations where N is the number of inputs and the number of state elements. Since this cannot be tested because the space is huge (excluding hierarchical designs and emulation), you generate targeted test patterns (and many many mutexes) to pare down the space, and perhaps randomize some of the targeted tests to verify you don't execute out of bounds. This is all true of software too. Certainly, it’s easier to test software, but I don’t think it’s because software state is combinatorially simpler than hardware state.
- deepspace 5y agoBack in the day, when I was cutting my teeth on embedded systems, I read an Intel Application Note - probably for the 8051. The section about managing watchdog timers stated that it is often good practice to deliberately let the watchdog time out periodically, at a convenient time. Then the system would be reset to a predictable state regularly, and failure states leading to unplanned watchdog timeouts would occur less frequently. To this day, all my systems which are intended for prolonged unattended operation reset themselves at least once a day.
- arcticbull 5y agoThe Payment Card Industry (PCI) standards that all credit card readers conform to mandate that they actually reboot at least every 24 hours :) [edit] More specifically there are a number of things that have to happen every 24 hours - some memory has to be zeroed, firmware integrity verified, etc. Most vendors like Verifone [1] implement this by rebooting the reader at least every 24 hours on a timer. [1] https://developer.verifone.com/docs/verifone-documentation/engage/global-payment-application/latest/rules/pci_reboot.html https://developer.verifone.com/docs/verifone-documentation/e...
- Akronymus 4y ago> at least every 24 hours on a timer. Shouldn't that be 'at most' or 'at least once'? (My English skills are lacking so that was quite the mental stumble for me)
- tomcam 4y agoYou are correct
- camtarn 4y ago'at least once every 24 hours' would be valid. 'at most every 24 hours' would imply reboots_per_day < 1 is bad and reboots_per_day >= 1 is good; i.e. that the standard was focused on preventing unnecessary reboots, but didn't care if the reader didn't reboot regularly, which I think is contrary to the comment's meaning. 'at least every 24 hours' is correct.
- surfmike 5y agoI found it interesting also that the Recovery-Oriented Computing project at UC Berkeley/Stanford puts "design for restartability" as a research area. http://roc.cs.berkeley.edu/roc_overview.html http://roc.cs.berkeley.edu/roc_overview.html
- drewcoo 5y agoSee also: crash-only https://en.wikipedia.org/wiki/Crash-only_software https://en.wikipedia.org/wiki/Crash-only_software
- B1FF_PSUVM 4y agoPiece featured recently here : https://news.ycombinator.com/item?id=8464573 https://news.ycombinator.com/item?id=8464573
- sam_bristow 4y agoApparently a ‘96 Toyota ECU can be restarted at 180 kph with only noticeable effect being a brief ignition ping. I've been told this isn't the case for newer cars though.
- tomcam 4y agoI’ll let you field test that one
- AdamH12113 4y agoOn a practical note, testing a system's ability to start up is usually much faster and easier than testing its ability to work properly for long periods of time. Real-world long-term behavior is unpredictable and somewhat chaotic, but there are only so many things you can do straight out of a reboot.
- josephg 4y agoThis is food for thought. I wonder if there’s some other ways we could architect operating systems so these problems wouldn’t manifest as much? Like, the “startup process” seems to be framed wrong to me. Why does it take tens of billions of clock cycles (seconds) to start up? Essentially all the system is doing is filling some memory with a known-good kernel image, and initialising some hardware devices. If we think about startup as “restoring a (deterministic) boot image”, then booting is much faster. And we might be able to use that to verify the kernel’s integrity periodically and maybe restore parts of that image at runtime if the system ends up in an invalid state. Sort of a “soft / partial reboot” process. In my testing code I make heavy use of check() methods. These methods go through all my runtime invariants and checks that they all still hold. This is fantastically useful in testing - a check method and a fuzzer do the work of 10x their weight in unit tests. I wonder if a similar method in the kernel (run every hour or something) could be used to find and track down bugs, and automatically recover when they’re found. “Oh the USB device is in an invalid state. Log and reset it!”.
- semitones 4y agoTo better understand why we can't just load a snapshot of the "desired" memory state directly from a cached image, we have to understand what the OS actually does during it's initialization. One of the main responsibilities of the OS is to manage hardware resources. During init, kernel drivers and modules perform health checks and determine availability and status of hardware resources, peripherals, I/O, etc. Since lots of things could go wrong with the various hardware components, at anytime, this process should be performed during each system boot.
- 4y ago
- twoodfin 4y agoIt’s been known for more than 35 years that most bugs found in production are “Heisenbugs”: https://www.cs.columbia.edu/~junfeng/08fa-e6998/sched/readings/why-do-computers-stop.pdf https://www.cs.columbia.edu/~junfeng/08fa-e6998/sched/readin... Gray’s paper provides a logical reason why: If a bug can be reproduced reliably it’s much more likely to be found during testing and fixed, so appearing seemingly at random is a kind of Darwinian survival adaptation.
- chii 4y ago> If a bug can be reproduced reliably it’s much more likely to be found during testing and fixed which means in order to find as many bugs as possible during development, the dev environment must match the prod environment as exactly as possible. Bugs don't "evolve" like a life form - unless you code is self-modifying! It is caused by a permutation of all possible inputs (where the environment is also an "input"). Limiting possible inputs (such as functional programming) means you limit the possible differences between dev and prod - so functional programming ought to produce less bugs!
- WoahNoun 4y agoOr go to the opposite extreme and use fuzzing.
- angry_octet 4y agoBugs don't evolve but the bug generators do. We build code with more and more complexity until we reach an equilibrium with our ability to debug the systems. If there is no competition in the evolution then systems can become arbitrarily baroque and fragile, which leads to rules like 'no software updates/patches' and 'all config changes via the CCB and pass mandatory tests' or 'your problem is handled manually', the wire wrap of systems development. Rebooting (the process or the computer) is trimming the state space. Model checking driven design really helps tame the tendency to let program state evolve without bound. FP tends to follow a bottom up design which makes this easier too.
- Akronymus 4y agoOr, if it is a system based on many components that can be independantly updated, some bugs may be introduced because a mismatch caused one variable to corrupt and then that one just keeps being propagated to newer versions until it triggers a bug because you assumed it can never have that specific value. Which, it actually can't in the current state, but only because of the system itself changin, or evolving, over time.
- omgJustTest 4y agoState management! Basically how can you know all the variables at one time? Start on initialization. IMO the fact that everything "works" after reboots is simply because the state is 1. well defined and 2. available to all developers.
- contingencies 4y agoAs it is now common to shut down subsystems for reasons of power optimization, this sort of approach may be becoming an accidentally autonomous architecture.
- nulluint 4y agoConsidering every animal sleeps on a daily basis, I would say “turning it on and off again” pertains to maintaining the stability of complex state-based systems.
- Vanit 4y agoAn interesting analogy, but it's worth remembering that we don't actually understand sleep as a mechanism very well.
- arthurcolle 4y agoI thought the analogy of defragmentation was considered acceptable. I stopped doing all nighters a few years ago and the days have become much easier to handle, even bad days. Always getting a good night's sleep is, in the modern vernacular of philosopher-poet DJ Khaled, a "major key alert"
- gerdesj 4y ago"Before a computing system starts running, it’s in a fixed initial state. At startup, it executes its initialization sequence, which transitions the system from the initial state to a useful working state:" LOL etc
- not2b 4y agoI remember attending a talk by Walter Bright maybe three decades ago, back when he was known for his C++ compiler for Windows, and before that his compilers for DOS. He talked about his very paranoid approach to assertions; in addition to a huge number of asserts for preconditions, he would start out with asserts essentially at every basic block; he'd take them out as he produced tests that reached this point. This was the DOS days and he didn't have a coverage tool, so the presence of an unconditional assert meant that this line hadn't been reached yet in testing. He said he developed that style because the lack of protection in DOS meant that any error (like a buffer overrun) could trash everything, right down to damaging the machine. He said that early on those asserts were enabled in the shipping code, making it appear that his compiler was less reliable than competitors when he felt it was the opposite, but I think he wound up having to modify his approach.
- leeoniya 4y agohe actively hangs out here, btw :) https://news.ycombinator.com/user?id=WalterBright https://news.ycombinator.com/user?id=WalterBright
- mise_en_place 4y agoEven with these approaches, you will inevitably run into Heisenbugs that creep up as runtime errors. Even the most powerful compiler in the world can't catch those. It can be mitigated by having a system that is traceable from start to finish.
- pvillano 4y ago
- Barrin92 4y ago"There seem to be certain analogies here between computing systems and biological ones. Your body is composed of trillions of compartmentalized cells, most of which are programmed to die after a while, partly because this prevents their DNA from accumulating enough mutations to start misbehaving in serious ways. Our body even sends its own agents to destroy misbehaving cells that have neglected to destroy themselves; sometimes you just gotta kill dash nine." I think that's also reminiscent of Alan Kay's philosophy behind OO in its most original form, and probably most closely realized in Erlang: "I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages (so messaging came at the very beginning -- it took a while to see how to do messaging in a programming language efficiently enough to be useful). - I wanted to get rid of data. The B5000 almost did this via its almost unbelievable HW architecture. I realized that the cell/whole-computer metaphor would get rid of data[..]" I wonder why so many of the most popular programming languages went into the opposite, very state and lock based direction given the strong theoretical foundation that computing had for systems that looked much more robust. http://userpage.fu-berlin.de/~ram/pub/pub_jf47ht81Ht/doc_kay_oop_en http://userpage.fu-berlin.de/~ram/pub/pub_jf47ht81Ht/doc_kay...
- ncmncm 4y agoAlan Kay has been for decades revising his description of how he thought. Suffice to say, Smalltalk-72 was not an OO language. That was bolted on later. (Which is not to claim that OO has, as Kay insists, any universal merit.)
- gjm11 4y agoI'm looking at the Smalltalk-72 manual right now and I see e.g. the following: > In Smalltalk, every entity is called an object; every object belongs to a class (which is also an object). Objects can remember things about themselves and can communicate with each other by sending and receiving messages. The class handles this communication for every object which belongs to it; it receives messages and possibly produces a reply, typically a message to send to another object. So far as I can tell, this means that Smalltalk-72 was what-Kay-considers-object-oriented. I'm not sure whether this refutes the claim you're actually making, because apparently you're saying that Kay lies about what he used to think, and maybe e.g. you're saying that what Kay now says about object orientation is not what he used to say, and that in the 1970s he wouldn't have considered Smalltalk object-oriented. Or maybe you're saying that the right way to think about object-orientation is something different from Kay's, and that Smalltalk-72 was not what-you-consider-object-oriented. Or something. Could you maybe be more explicit? Could you give some examples of things that Kay now says he used to think, and explain why you believe he didn't actually think them? Could you explain in what sense you reckon object-orientation was absent in Smalltalk-72 but bolted on later, and why you think that indicates intellectual dishonesty on Kay's part?
- sodiumjoe 4y ago> A novice was trying to fix a broken Lisp machine by turning the power off and on. > Knight, seeing what the student was doing, spoke sternly: “You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.” > Knight turned the machine off and on. > The machine worked. from http://www.catb.org/jargon/html/koans.html#id3141171 http://www.catb.org/jargon/html/koans.html#id3141171
- Banana699 4y agoI don't get it.
- gjm11 4y ago(Tom) Knight says that fixing problems requires you to understand what's going on, and you can't just blindly turn a machine off and on again and hope it'll fix things. In the novice's particular case, Knight understands what is actually wrong with the machine, and that in this case it will be fixed by turning it off and on again, so he does that. The idea that in such a situation the machine wouldn't be fixed by power-cycling it when the novice does it, but would when an expert with deep understanding does it, is a joke. The joke is mimicing the form of a Zen koan. Koans often play with contradiction, I think with the idea of shaking the reader out of simplistic black-and-white thinking into something more holistic and less dichotomizing. I don't think "you need to understand things deeply and not just make random easy changes in the hope of fixing them, but sometimes it happens that those random easy changes are what's actually required" really counts as the sort of holistic non-dichotomizing thing Zen is trying to teach, but it's kinda in the right ballpark.
- colinprince 4y agoSee also http://planetvermont.com/v9n2/machine.html http://planetvermont.com/v9n2/machine.html
- 0xbadcafebee 4y agoWell yeah, all modern computers and software are nondeterministic state machines. The order and set of operations is nondeterministic, and its input, output and algorithm constantly mutates. As there is no function within the computer to automatically reverse its tree of operations to before it encountered a conflict, an infinite number of state changes inevitably leads to entropy, thus all running systems [with state changes] crash. Since the computer is a state machine, and all initial state is read from disk, restarting the computer reinitializes the state machine from initial state. Rebooting resets the state machine to before state mutation entropy led to conflicts. Rebooting is state machine time travel. We could prevent this situation by designing systems whose state and algorithms don't mutate. But that would require changing software development culture, and that's a much harder problem than rebooting.
- legalcorrection 4y agoIt is 100% theoretically possible to build a complex system that never enters an invalid state. But because human beings are fallible—and attempts to correct that infallibility are not just expensive, but themselves subject to further fallibility—no such complex system will ever be built.
- mkleczek 4y agoActually building such a computer is provably impossible: https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem I highly recommend this talk on the subject: https://pron.github.io/posts/correctness-and-complexity https://pron.github.io/posts/correctness-and-complexity
- legalcorrection 4y agoRice’s theorem says no such thing. Rice’s theorem tells you you can’t necessarily prove you’ve built a flawless system, not that you can’t build a flawless system. You can build a perfect program even if you can’t necessarily then know you have. Perhaps you interpreted my post to suggest that you could build a machine that verifies that any arbitrary software run on it never enters an invalid state? If so, I agree, that is not possible. I made a minor edit to make it clear what I’m talking about.
- logbiscuitswave 4y agoI once heard someone use the term “therapeutic reboot” for when computers get into a bad operating state. It really stuck with me.
- ncmncm 4y agoWe thus internalize our oppression, and become our own abusers.
- arcticbull 4y ago"Not a single living person knows how everything in your five-year-old MacBook actually works. Why do we tell you to turn it off and on again? Because we don’t have the slightest clue what’s wrong with it, and it’s really easy to induce coma in computers and have their built-in team of automatic doctors try to figure it out for us. The only reason coders’ computers work better than non-coders’ computers is coders know computers are schizophrenic little children with auto-immune diseases and we don’t beat them when they’re bad." [1] [1] https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks
- TedShiller 4y agoWorks great for bad OS’s like Windows
- herbst 4y agoNo idea why you got downvoted. As a Linux user I don't restart my whole computer to try fix something. I can't even remember the last time I restarted without having installed a new kernel. I also don't know why I would, I can restart the broken components and get a less cluttered debug log when doing only that.
- TedShiller 4y agoSame, my FreeBSD uptime is approaching 1 year. Server, not good for desktop tho
- germinalphrase 4y agoWill we never be done with this “unreasonable effectiveness” cliche - or am I missing something?
- YPPH 4y agoIt's an unreasonably effective cliché.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- brenainn 4y agoNot until it's considered harmful.
- Akronymus 4y agoI find it quite sad how pervasive the 'considered harmful' line is. It wasn't even the original title of the article. 'A Case Against the Goto Statement' was a much better title.
- IshKebab 4y agoHN needs a blanket ban on that and "considered harmful", and "rise and fall", and "for fun and profit". People are so uncreative. Maybe we can make a browser extension that rewrites bad titles.
- cvoss 4y agoI would go further and call it a meme at this point. And it's completely lost its effectiveness. After all, this article uses the phrase to mean exactly the opposite of what it originally meant, arguing that it is, in fact, _perfectly_ reasonable to expect that rebooting will solve your problem: > I offer the following argument that restarting from the initial state is a deeply principled technique for repairing a stateful system [emphasis original] Whereas, Wigner's original article on the "Unreasonable Effectiveness of Mathematics in the Natural Sciences" has this statement: > The miracle of the appropriateness of the language of mathematics for the formulation of the laws of physics is a wonderful gift which we neither understand nor deserve.
- vivegi 4y agoA possible analogue of this for building/packaging systems is the DOCKERFILE for building a container for infrastructure. No wonder, it acts as a reliable state initialization procedure.
- hinkley 4y agoI am utterly in awe every time there's a breakthrough in durable random access memory, where someone (often the author) is so very excited about how this will mean we will never have to power off our computers again. Flip a switch and it's instantly on! Have you lost your ever-loving mind? Of course you're going to reboot. Bit rot improvements have only ever been incremental. To a first order approximation we've expanded from a day or two to a couple of weeks/months for general purpose computers (I argue that servers are not GP, and so their spectacular uptimes don't translate). In thirty years, that's barely more than an order of magnitude improvement. You're gonna need a couple more orders than that before taking away the reboot option sounds anything like sanity. If you can keep a machine from shitting the bed for 2-3 times longer than the expected life of the hardware, then we can talk, but I'm not making any promises. Until then I'm gonna reboot this thing sometimes even if it's just placebo.
- drpixie 4y agoQuite a bit of our infrastructure runs for very long periods (long in IT terms) - years between faults or reboots. Sure lots of domestic modems & routers have to be reset often, but it's common to find infrastructure routers that are only reset when they have to be moved! Says a lot about the software in domestic equipment. I've worked with a mainframe that ran for a decade, and was only rebooted when a machine room fire required a full powerdown "right now"! But people are impressed when their Windows stays up for a few day %#%$#@#$!! And remember Voyager, been ticking along since 1977. But all these impressive systems are built to be impressive. Probably error correcting RAM, etc. Shows we can build reliable hardware. And if a substantial fraction of users wanted it, things like ECC RAM etc would be only fractionally more expensive than our error prone alternatives. Of course,
- zeusk 4y agohow much of that hardware can run a random executable from the web? and does so pretty often?
- dieselgate 4y ago
- nesarkvechnep 4y agoBefore HN, I didn’t know so many things are unreasonably effective.
- usrbinbash 4y agoThere are 2 hard problems in IT: 0. Cache invalidation 1. Naming things 2. Off-by-one errors Index 0 is why reboots are so effective.
- mro_name 4y agoI find even leaving them off pleasing. Don't run compputers 24/7, your brain doesn't, too. The smaller the scale (amount of people affected), the more sense do business hours make. Ot times of non-operation. Who cares if a personal/smallbiz webserver sleeps for some hours at night?
- KronisLV 4y agoSo what's everyone's take on restarting servers every now and then? I think that it might be something that you do occasionally (say, once a month or so) for each of your boxes, regardless of whether it's a HA system or not. Though perhaps I can only say that because the HA systems/components (like API nodes or nodes serving files) won't really see much downtime due to clustering, whereas the non-HA systems/components (monoliths, databases, load balancers) that i've worked with have been small enough for it to be okay to do restarts in the middle of the night (or just weekends or whatever) with very little impact to anyone, even without clustering at the DB level/switchovers. I mean, surely you do need to update the servers themselves, set aside a bit of time for checking that everything will be okay and don't try to patch a running kernel most of the time, right? I'd personally also want to restart even hypervisor or container host nodes occasionally to avoid weirdness (i've had Docker networks stop working after lots of uptime) and to make sure that restarts will result in everything actually starting back up afterwards successfully. Then again, most of this probably doesn't apply to the PaaS folk out there, or the serverless crowd.
- gjulianm 4y agoThe more time it passes, the less need I see for periodic restarts. For example, a few years ago I would need to restart some servers due to memory fragmentation that would cause some allocations to fail (we needed some processes to get big chunks of continuous memory). Right now it doesn't happen. Memory leaks are rarer, and in general things work well. I have some servers with more than a year uptime and still working well.
- DeathArrow 4y agoAnyone old enough to remember IT Crowd? Here's my favorite quote from it: https://www.cipher-it.co.uk/wp-content/uploads/2017/11/ITCrow.jpg https://www.cipher-it.co.uk/wp-content/uploads/2017/11/ITCro...
- Yizahi 4y agoAlso designing something "crash-first", even if you don't call it like this, leads to many different approaches and possible improvements. For example lets imagine some embedded device in the ISP network, it is not very accessible so high reliability is required. You can overengineer it to be a super reliable Voyager-class computer, but that a) will cost much more money than it should, b) you will fail to achieve the target. Or you can go crash-first approach. Many things can be simplified then, for example no need to have stateful config, no need to code any config management which saves it, checks it etc. You just rely on receiving new config every boot and correctly writing it to controllers. Less complexity. Then if you are crash-first you expect to reboot more often. You then optimize boot time, which would be a much lower priority task otherwise. And suddenly you have x times better (lower) downtimes per device. You can optimize on some hard stuff - e.g. any and all 3rd party controllers with 3rd party code blobs and weird APIs. Instead of writing a lot of health checks for each and every failure mode of the stuff you can't really influence, you write a bare minimum and then watchdog to reboot whole unit and hope it will recover. And this works very well in practice. The list goes on. Instead of very complicated all-in-one device you have a lightweight code which has a good and predictable recovery mechanism. It is cheaper, and eventually even more reliable than overengineered alternative. Another example - network failure. Overengineered device will do a lot of smart things trying to recover network, re-initialize stuff, re-try different waits (and there will be a lot of re-tries), and may eventually get stuck without access. Lightweight device has a short simple wait with simple retry, or several, and then reboots. Statistically this is better than running some super complicated code, if the device is engineered to reboot from the start.
- shadowofneptune 4y agoYou can see this approach even in spacecraft. The Apollo Guidance Computer was an example of a crash-first system (https://www.ibiblio.org/apollo/hrst/archive/1033.pdf https://www.ibiblio.org/apollo/hrst/archive/1033.pdf).
- Banana699 4y agoMandatory Light Years Ahead | The 1969 Apollo Guidance Computer https://www.youtube.com/watch?v=B1J2RMorJXM https://www.youtube.com/watch?v=B1J2RMorJXM 34C3 - The Ultimate Apollo Guidance Computer Talk https://www.youtube.com/watch?v=xx7Lfh5SKUQ https://www.youtube.com/watch?v=xx7Lfh5SKUQ
- albertzeyer 4y agoOn an application level, this means to restart the application. Even when no unexpected exception occurs, sometimes frequently restarting the application can be a valid solution. For example when there are potential memory leaks (maybe in your code but maybe also in external library code) and the application should just run infinitely. Btw, a simple method to restart an application: https://stackoverflow.com/questions/72335904/simple-way-to-restart-application https://stackoverflow.com/questions/72335904/simple-way-to-r...
- iainmerrick 4y agoOne of my first jobs was working on DAB digital radios. Our workhorse low-end model was quite flaky, but it had a watchdog timer that would reboot it if it hung; and it was so simple that it would get back to acquiring a signal in a fraction of a second. So the upshot was, a system crash just meant the audio would cut out for maybe 6 seconds, then start playing again. For the user, completely indistinguishable from poor radio reception, which is always going to happen from time to time anyway. Crashing wasn’t a problem at all as long as it was rare enough. (I like making robust software and hate crashes, so I don’t like that approach, but I use that experience to remind myself that sometimes worse really is better, or at least good enough.)
- fennecfoxy 4y agoI don't think it's that unreasonable for a complex system with complex states to reboot. I imagine (I'm not a medical professional) that it works in roughly the same way as when human beings black out; reboot.