5 ms·
I was stuck in the planning phase for a pet project up until a week ago, for a reason why Lisp and Smalltalk always made me nervous. Bugs can put your persiste
by hinkley 6y ago
I was stuck in the planning phase for a pet project up until a week ago, for a reason why Lisp and Smalltalk always made me nervous.
Bugs can put your persistent state into uncharted territory, and there may be no clear path back. There's a reason we still have 'turn it off and back on again' in our bag of tools. Often it's the only thing that reliably works. This makes systems designed never to be shut down and turned back on again deeply unimpressive to the more practically minded members of the audience.
One of the places persistent state crashes and burns is when the system of record and the source of truth combine. Once the source of truth breaks we can't know what's true anymore.
My breakthrough was figuring out that I'm trying to get data from something that is already a pretty good system of record, I can just keep letting it do that job indefinitely. My source of authority needs to transform that data, not own it. As long as I can detect out-of-band changes to the system of record (which I can), I can always rebuild my models from scratch. That allows me some excellent test fixtures. That also gives me the option to do manual 'surgery' on the system of record rather than sinking my roadmap to implement a full feature to do something I might not need to do again for another year, or that no customer will ever see.
I am just on the same continuum as everybody else. My code needs to make a lot of decisions. I can't afford to make them all from first principles every time. I need to store intermediate values. I also need to identify intermediate values and question them. If I don't get help with this from my architecture, myself and coworkers will blur the lines every time a problem seems a little beyond our abilities, and eventually nobody will know what's true anymore except the delusional ones. How do I know this? Because I have never seen any other outcome. The only differences are in how much surprise the team exhibits.
- klodolph 6y ago> There's a reason we still have 'turn it off and back on again' in our bag of tools. Often it's the only thing that reliably works. An alternative perspective—at the very large scale, the opposite may be true. Turning something off and then on again may be one of the worst ways to fix problems with certain large systems. At a large scale, you would document and manage the state of the system, make changes which put that system into a new state, and build tools that monitor and enforce system state (declaratively where possible, but it’s not always possible to be declarative). Sometimes there will be duplicated effort between making sure a system runs smoothly and making changes while it’s running, and making it so the system can turn off and on again. Think about the duplicated effort between setting up a new database and making schema changes to a live production database. YES, you want to have a source of truth. But the “turn it off and on again” source of truth is usually a sequence of imperative commands rather than a description of the desired system state, and comes with its own reliability problems. A description of system state as source of truth has a different set of reliability problems but I think it’s absolutely necessary to explore the solution space between these two extremes—shooting for a completely declarative system is futile, but an imperative system can be very hard to reason about.
- guidoism 6y agoYeah I think the whole “need to reboot to get to a clean state” is maybe a bug not a feature. It is certainly possible to build systems that can run for years or decades. I’m guessing that persistent memory will require us to think about changes to state in memory are more like ACID in databases. We should think about rollback instead of reboot.
- hinkley 6y ago> is maybe a bug not a feature. Once you start thinking of safety as a feature you've ruined your ability to think responsibly about anything else. A reset is a safety mechanism. You shouldn't need it, but only a sociopath would try to get rid of it. Almost all of the software we write is now affecting people's lives. That your bug may eat someone's term paper has been a problem for decades, and nobody ever internalizes how that affects that person's life. The avalanche of cause and effect may never settle down. Now we can get people fired. Ostracized. We are trying to take the power but not the responsibility, and it's bullshit. No, hypocritical bullshit, because we spend a lot of time here complaining about others doing the same thing.
- klodolph 6y ago> You shouldn't need it, but only a sociopath would try to get rid of it. > Almost all of the software we write is now affecting people's lives. Don’t twist a technical disagreement into some kind moral outrage. We both have the same goal of providing reliable software to people. The problem is that for the past decades, resetting the machine has deleted countless term papers. Nowadays, people generally put their term papers in cloud services that are always on. This is a huge step forward. We can bring this kind of reliability to offline usage. I think this requires taking lessons learned from cloud/server farm/mainframe world (always-on) and figuring out ways to apply them to personal computers.
- sizt 6y ago... a new OS and let's call it QAnon !
- ww520 6y agoWifi routers have this notorious problem with the nvram. Sometimes some state in the nvram is corrupted and the routers would stop working. Turning off and on again won't work. A factory-reset to wipe everything out is the only way to get out of the corrupted state.
- hinkley 6y agoAround the time of Randall Monroe's "shibboleet" comic, I had an ISP connection that would flake out. If nobody kicked the cord on my end, the same series of events played out every time: Check the cables. Turn it off for 30 seconds. Check the connection to the router. Check the lights. Try random other shit. ISP reboots their modem instead of mine. Networking restored. Now, in theory, when a commercial modem loses carrier it should reset the connection completely. This they kept telling me, every four or five months for two years. And every time I got to sit patiently while their support person learned that this is not always the case, over the course of the next 20 minutes. If I tried to tell them about last time, then they would verbally pat me on the head and keep wasting my time. Whatever cooperative game or The Killer Is About to Reveal Himself was now a distant memory.
- monadic3 6y ago> This makes systems designed never to be shut down and turned back on again Not necessarily; you just need a way to bootstrap the memory. Plan9 is pretty easy to run a fully functional terminal from memory/network access. Live CDs work pretty well, too.
- jandrewrogers 6y agoAs systems become very large, ‘turn it off and back on again’ is often counter-productive to the point of being the most destructive thing you can do. Repairing state is slow and expensive, and meanwhile the world is happening. This is the reason why all ultra-scale systems use some kind of “k-safety” durability model. The system would effectively be unrecoverable if you actually rebooted it because it would struggle to get back online. This is increasingly the rule rather than the exception for data infrastructure. Even in the handful of recent systems I worked on where there was an explicit external system of record and rebooting was an option operationally and technically achievable, we still had to do semi-exotic custom engineering to make sure it was practically achievable. When I started in the industry, ‘turn it off and back on again’ was the default strategy. These days it is rightly reserved for the dwindling number of cases, outside of small systems, where it still works. The computational cost of restarting is prohibitive.
- simonh 6y agoRight the problem is really about maintaining consistency, how do you checkpoint the entire system and all your different components with their derived values. I used to work on a system with a distributed hierarchical object store. We had copies of the data in various geographic regions, with copies of the same or different applications running in each region connected to a local copy of the store. When you write to your local copy, the storage system would propagate changes out to the other copies. Maintaining consistency was a constant problem. One way to handle it was to give each region running a copy of a given application it's own regionally named folders in the object store. The problem is similar to when you have application or database clusters with nodes that lose contact with each other, so called split brain syndrome, and strategies for determining which node should become authoritative. This is where the term STONITH comes from - Shoot The Other Node In The Head. You might not want to restart the whole system, but you do need to be able to tell subsystems that they need to refresh their state. A reboot is simply a manual implementation of that, but in a continuously running system you need to build algorithms and strategies for deciding when to do such refreshes in the system itself.