23 ms·
Simple Systems Have Less Downtime
- blacksqr 7y agoEvery new programming style epiphany is an ad hoc, intuitively-specified, hazy restatement of the Unix Way.
- deleted 7y ago[deleted]
- testuser66 7y agoAnd thus, a new law was born.
- mhd 7y agoIf Unix is your yardstick for simplicity, that just shows how far we've come…
- ailideex 7y agoLinux is incredibly simple compared to Windows once you actually your computer as a computer instead of as a really poorly made gaming console.
- vntok 7y agoTry talking Grandma over the phone to help her share her photo directory with Aunt Jeanine, on the same PC. No command line allowed, and you know she can only focus on that type of task for 4, maybe 5 minutes top.
- ptx 7y agoI can't imagine any system where you'd have a good chance of succeeding in that scenario without some kind of remote administration tool. Remote controlling a user over the phone is difficult, unreliable and time-consuming.
- lallysingh 7y ago.. install remote admin tool? Gotomypc etc? Or just tell them to copy it to a USB stick. Yep, same machine, it's still the best answer.
- t0astbread 7y agoShare? Google Drive. Or email. Both of which are equally good/bad to explain to grandma on Windows and Linux.
- userbinator 7y agoNo command line allowed It's easier to tell someone what keys to press than to find and manipulate UI elements. on the same PC On a PC, everyone already has access by default, so "sharing" doesn't involve anything additional.
- mhd 7y agoThat wouldn't work reliably under Windows, either, as nobody knows anything about files and directories generically anymore. (That's the reason why first the Windows explorer was suddenly sufficent enough and then almost completely disregarded) A public folder on the same PC is way harder to explain than just telling your photo app to share it with your resident spying company/cloud provider, despite how wasteful the round trip is. Not that that has anything to do with a discussion about the "unix way" as a special case or even superset of IDEs.
- darkmighty 7y agoMy mother can't reliably remember how to operate the remote control but uses linux just fine (browsing, editing text documents, etc).
- SahAssar 7y agoI think you're confusing easy and simple. Windows is easier (for most people), but linux is certainly simpler.
- AnIdiotOnTheNet 7y agoBullshit. You're talking about an operating system where the common advice for someone who wants to install up to date software is to fucking compile it from source because the whole community never got their collective shit together enough to allow developers to directly distribute binaries without a gigantic fucking headache. Christ, it's such a fucking mess that one of the most compatible ways to distribute software is to write it for Windows and rely on WINE.
- busterarm 7y agoHardly. I've maybe had to do that 5 times in the last...10 years? And I split my time between Arch, CentOS & OpenBSD. The vast majority of things these days are packaged in a useful way. There's also flatpak and similar now.
- AnIdiotOnTheNet 7y ago> I've maybe had to do that 5 times in the last...10 years? Good for you. Some of us have to install up to date software more often than every 5 or 10 years.
- busterarm 7y agoI'm running Arch on my office workstation. I install up to date software literally every day.
- AnIdiotOnTheNet 7y agoOnly after a third party makes an AUR for it. I'll stick to an OS where the developer can publish directly to the users without 15 different packaging formats, thanks.
- ailideex 7y agoAgain, flatpak, appimage, docker ... you can use one of these, don't have to use 15.
- joana035 7y agoUNIX is very simple, it just needs a genius to understand its simplicity. And it is 2020, I guess nowadays everybody understand how unix-like system works and how simple they are.
- beigeoak 7y agoI disagree. I have an entire compressed text file on the simplicity of the Unix way. Once I remember the tar flags to extract it, I'll post its contents here.
- carlmr 7y ago>For example, an analytics dashboard built with a no-code analytics tool like Looker is likely to have more qualified people to fix it than one built with a patchwork of custom scripts and APIs. Nobody should have to pull data scientists or product developers away from their work to fix a bar chart. I don't have experience with Looker, but the unseen complexity of no-code tools often leads to very complex systems with interactions that are hard to understand. A dashboard may not have that many interactions with other tools, but the black box nature of these kinds of tools usually leads to significant downtimes when there's a small detail not working as expected, or where you need a small customization that wasn't foreseen by the no-code API creators.
- buremba 7y agoThat's true for drag&drop tools that provide a user interface as an alternative but Looker or similar tools introduce their own language that limits your capability by sandboxing the environment but still lets you programmatically define your models. See LookML for Looker (https://docs.looker.com/data-modeling/learning-lookml/what-is-lookml https://docs.looker.com/data-modeling/learning-lookml/what-i...) or Jsonnet for Rakam. (https://docs.rakam.io/docs/compose https://docs.rakam.io/docs/compose) P.S: I'm affiliated with the company Rakam.
- jbott 7y agoI think that’s missing the point he was making. No matter what front-end representation of a system you get, the implementation backing a no-code system will have edge cases around interactions both internally and with third-party systems. The fact that you’re hiding a system behind a veneer of “no code” doesn’t hide the fact that code is behind the actions that are taking place, it just means you can’t introspect it (even if this “no code” is in fact code). I think a better point would be to make that that’s not a bad thing, sometimes abstractions provided in “no code” systems can greatly simplify a solution. But even the most well-designed solution will fail to some essential complexity of the root-problem the original authors didn’t understand. Without debugging tools (or privileged access to the backend of the implementing system) it’s difficult or impossible to understand what went wrong.
- 7y ago
- winrid 7y agoI love simplifying things.
- nineteen999 7y agoI'm not sure I follow, could you break that down for me?
- monus 7y agoUsually, being easy is confused with being simple. Easy to use systems usually tend to be very complex but they'd only hide it from the average user. Once you go one step ahead, do something outside of quick start, you find yourself helpless. In that sense, the article is on point. Not only the management of the ship is simple, machinery behind the scenes is simple as well. This applies to programming languages, too. Python is easy but when you do advanced stuff, often without realizing, you start to grok many complex details to make it work or make it work in a performant way. On the other hand, Go (and Rust supposedly, but I don't have experience) is simple. It doesn't let you do many things and that results in fewer ways of doing a specific task but when you do it, you feel safer because in the end, it does look and feel simple although you spent 1+ hour than you'd normally spend compared to coding in Python (this assumes you're doing some advanced stuff, not quick start).
- tsimionescu 7y agoGo is actually a pretty complex language, more so than Java for example, in my opinion. It has all sorts of primitives that you need to get used to, special rules for how the built-in types work, there are all sorts of rules to see what creates a copy of a struct and what doesn't, pointers have at least three completely different use-cases (optionality, mutability, and avoiding copies), several rules for what can and can't be addressed etc. And this is all before looking at the standard library.
- monus 7y agoI think concrete things you listed like the case for pointers is not much different than Java. While why you use a pointer changes, it's simply pointer in the end, just like Java. I'd say Go is able to put so much meaning to those fundamentals because 1. they have to, there isn't many other mechanisms 2. they can because it's still early in the evolution of the language. After some adoption and wildly different use-cases that users wants to be addressed, those patterns start to disappear and people end up with lowest common denominators, pointers being just pointers in this case.
- XorNot 7y agoI do wonder what the trade off would've been in Go if something like: a := b was guaranteed, under all expressible circumstances, to create a completely new copy - regardless of the content of b, unless it was explicitly a pointer. It does frequently feel like this could've been solved by making basic slice assignment always go as a := b[:], and adding some type of syntax and checker where you needed to confirm that you were doing a not-completely immutable copy.
- vasergen 7y agoThe problem is that for different people 'simple' has different meaning.
- uk_programmer 7y agoThis feels like it is stating the obvious. However it is easily forgotten. A lot of Software Engineers / Developers in my experience tend to over engineer projects or they are mandated they do so by so their dev leads. It is easy to get caught up into abstracting almost everything away almost for the sake of it while it provides no real benefit.
- hnick 7y agoDon't forget the old classic Resume Driven Development.
- abinaya_codes 7y agoI love simple things, thank you for sharing!
- stormdennis 7y agoHis website had this annoying message about subscribing for updates that has no obvious way of getting rid of it excepting by clicking the accept button. I guess that's simpler than having an additional "no thanks" button. The article was a formulaic retelling of well known wisdom too.
- gk1 7y agoHey, there is an X in the corner to close it. The subscription slider is made by Mailchimp so I don't have much control over the size of that X, unfortunately.
- vezycash 7y agoUsing the company this author helped to migrate to hubspot as an example. I'm not sure if the simpler solution - hubspot existed when the company started hacking solutions together. Also, the move to hobspot was relatively easy because the business processes and workflows were already well known and defined. Starting from scratch, fighting the fires as they appear, a bit of patchwork seems unavoidable. Everything is obvious in hindsight. I guess the real lesson here: Simplify once in a while. Like Facebook just did with messenger.
- notyourday 7y ago> I'm not sure if the simpler solution - hubspot existed when the company started hacking solutions together. Author's value proposition is "I will help you outsource your complicated marketing tools". If you reject the gospel that your marketing tools are complicated, author's service would be not needed.
- gk1 7y agoNot the intention at all. My job is to parachute into a startup facing some problem, fix the problem quickly, and get out. I have no business being at a startup that's running perfectly, or creating solutions that will require me to stick around.
- ThePhysicist 7y agoI don’t know, most systems like the container ship he/she described just seem simple but are actually pretty intricate inside. The logic behind them might be easy to understand, but the devil is in the details. A combustion engine is also simple conceptually: make stuff explode in a closed chamber to move a rod, convert that the movement to a rotation of a shaft. Super simple, yet I doubt I could debug many problems my car engine has with such basic knowledge.
- XorNot 7y agoThis seems like an oddly appropriate metaphor though - engines aren't super-complex once you know what everything does, but the biggest problem in car mechanics is just getting access to the specific part you need to measure or look at.
- xiaodai 7y agoDo they also have less people using it?
- speedgoose 7y agoIt's important to find a compromise. The simplest system at my work would be something like a php/mysql backend hosted on a old fashion shared server using ftp. Good luck attracting good developers with that. We use something over engineered but we have fun. The customers don't care but we do.
- collyw 7y agoThat's all good if your company is making money.
- Double_a_92 7y agoSince PHP has become more decent with the last versions, if you are starting a new project that maybe actually not even be too terrible.
- somurzakov 7y agoThere are a lot of companies where Senior PHP devs are paid more than Senior Java devs, because it is so hard to find a brilliant PHP dev
- DecoPerson 7y agoI like to “embrace complexity.” Any line of code, especially one that introduces a new concept, like a class or a method, is in “addition” to the root problem we’re trying to solve. The closer the code/data/org-processes/whatever matches the fundamental problems we’re trying to solve, the less complexity we introduce. However, you cannot “remove” complexity! A common mistake I see in software companies is using a single Jira ticket for one or more reports, one or more conclusions, and one or more action items. For example, if we get reports A, B and C, they may all go into a single Jira. Oh, but C is a totally different thing! It’s not a “duplicate”! Ok now someone needs to extract the info from a comment into a new Jira (which no one will... and it’d be a mess if they tried). Now a bunch of discussions are happening in the comments of this mega-Jira. 3 conclusions come out of these discussions — perhaps what the cause is and what should be done about it. Then let’s say there’s two action items, one is a temp fix and one is a long term fix. Unfortunately, I usually see people is the same Jira for both tasks (“Assign this back to me when the temp fix is in so I can do the real fix”). But splitting Jiras up is frustrating. The actual splitting is frustrating to set up, and it’s a pain to browse them. So most Jira workflows remove fundamental complexity (reports vs discussions/conclusions vs tasks) and introduce extraneous complexity (tonnes of fields and workflows that don’t necessarily apply). One should “embrace” the fundamental complexity of the problem(s) they’re dealing with.
- silon42 7y agoAgreed. It should be almost as simple to create a sub-task (of any task) as it's to comment. Also convert a comment to sub-task. Browsing is another problem...
- TeMPOraL 7y agoAnd there a fundamental impedance mismatch between these tools and actual work rears its ugly head. Work is structured as a dependency graph - a DAG. To complete C, you need to do A and B, but a part of B depends on a part of A to be done, etc. Yet these tools insist on a list, or a very flat tree (a task and maybe a subtask, but no sub-subtasks).
- jyounker 7y ago
- hyperion2010 7y agoThe only language that I have worked with that realizes that in the real world simple is _always_ a lie, is common lisp. It is the only language that has actually embraced the fact that its designers/committee were not geniuses and provided the tools for dealing with the complexity of the system. When unix tools fail, hope that you are on a system where it is possible to get the symbols and/or the source code, and even then good luck fixing things in situ. Most existing systems do not empower the consumer of code to do anything if it breaks. CL remains almost completely alone in this, because the debugger is part of the standard. Most of my code is written in python, and I can tell you for a fact that when python code really fails, e.g. in a context with threading, you might as well burn the whole thing to the ground, it will take days to resolve the issue, roll back the code and start over. The fact that people accept this as a matter of course is pure insanity, or a sign that most programmers are in an abusive relationship with their runtime environment.
- pjmlp 7y agoForgetting Smalltalk here? Mesa/Cedar, Oberon System were also like that by the way. Java and .NET also share a bit of it, after all Java is half-way to Lisp (as per Guy Steele), and .NET follows up on it.
- hyperion2010 7y agoSort of. I have a bad habit of forgetting Smalltalk in these kinds of conversations, but that is possibly because Smalltalk images exist in their own happy little worlds. I love working in Smalltalk environments, it is always a mind blowing experience, the system is completely homogeneous, accessible, introspectable, modifiable, etc. -- until you hit the hard boundary between the image and the host system, be it hardware or software. That boundary leaves quite a gap that smalltalks tend to be unable to fill by themselves (mostly due to a relative lack of resources). I can imagine a smalltalk system that could go all the way down to modern assembly, but unfortunately no such system exists today (that I'm aware of). If you can live inside the image then yes, Smalltalk is possibly even better.
- cal97g 7y ago
- buboard 7y agothe article is self-refuting in a way, although the message makes sense. The author describes simplicity as "fewer levels of abstraction, and each of them well understood" , like a ship rudder (which is a good definition). But no-code tools etc are anything but well understood, and usually introduce hundreds of levels of abstraction (due to their genericality). The same goes for frameworks that introduce tons of abstractions in order to cover cases that are irrelevant to the problem at task. Ideally, the chain of scripts that do one thing and can be read in 10 minutes is actually simpler and a much better way to fix the ship.
- praptak 7y agoThis holds up to a point. The systems that require really little downtime get progressively more complex. Need to handle machine failures? Push a bugfix without causing downtime? Maintain a single state across the redundant machines? Handle load spikes? Ones created on purpose to DOS you? All of this requires additional mechanisms, which can themselves cause failures.
- vasco 7y agoA bunch of paragraphs to type out KISS.
- sebringj 7y agoIn terms of the little guy having a saas, this is true for me...my saas basically is relying on s3 for both file storage and a json file per account to list out users allowed in the system. The only time it goes down is when s3 goes down or the occasional container restart.
- dhbradshaw 7y agoI'd be interested in hearing more details here. Are you saying that you use s3 as your only data store?
- sebringj 7y agoYes...think of it as layers... or manifests that point to another set of manifests global (list of users) -> accounts (have lists of users in them and role), content -> list of json files. I use a combination of json logic or simply listing out keys in a specific bucket...the speed of access doesn't seem to be something that bothers anyone, it started out as an experiment of less is more and because the app is not heavy on db searching etc rather only specific to account. I use oauth2 instead of relying on my own infrastructure and let people send out invites where they would use an email only but it must support the oauth 2 providers i already have in place. Basically, doesn't go down because it relies heavily on other things working that has teams of people working on them with gigantic resources. Don't get me wrong...I'm not saying it will scale to a million users, my clientele is mainly corporate that have a good amount per account sale but if it needs scaling, at least i didn't waste time and money on it during the interim and can always db-back it or dynamo it later.
- z3t4 7y agoSometimes the complexity is there for a reason. But if you can just start over, those reasons was not that important!? So don't hang on to functionality because it would be nice to have, you will have to pay the price for those features over and over.
- collyw 7y agoSometimes, but a lot of devs fetishise complex stuff. I disagree wit a lot of my tech leads decisions because he always wants to do the "most correct" way, whereas I want to keep things as simple as possible with the aim of keeping our code maintainable. For example, he wants to a add a graph database to the tech stack, while I realise that its one more thing to manage and potentially break and mongodb (that we already have) will be probably be good enough.
- z3t4 7y agoThings only get complexed if you mix them, like cords in a box. So you might be able to keep it simple as long as you do not have code that depends on both the databases, and that the two databases don't depend on each other.
- unnouinceput 7y agoVery good article. Please also preach this article to those who praise Kubernetes right and left (remember, Kubernetes it's the new blockchain hype; and just like blockchain you don't really need it unless you're a cryptocurrency; which 99% of companies who use Kubernetes aren't).
- Taek 7y agoIt's similar to optimization in that it follows a U-curve. Up to a point, simpler things have more performance, and simpler things have less downtime. But after a certain point, it's no longer enough to make it simpler. You have to get a lot more involved again, and this time the complexity has to be intelligently focused around the end goal (be it performance or uptime).
- dbg31415 7y ago"[The] more simple any thing is, the less liable it is to be disordered, and the easier repaired when disordered..." - Thomas Paine, Common Sense, 1776
- pkolaczk 7y agoI like the ship analogy from the article. The fact it is so robust is not because the system as a whole is simple, but because its components have well defined, narrow responsibilities and there are abstract interfaces between them, which hide a lot of complexity. You don't need to understand the internals of the pump to explain what it does, even though the actual internal technology may be quite complex and clever. Also you don't need to understand the internals of the pump to design a rudder. This is a property that many IT systems lack. Insufficient abstraction leads to a network of components where you can't reason about anything without understanding all of it. And such systems often break in weird ways.
- notyourday 7y agoA startup is not a container ship going between port A and port B where everything is known. A startup is a new destroyer that has been floated, did not have sea trials and went to war with a crew that might have a couple of people that used to a drive container ship but mostly staffed with kids that thought it was cool to play with a destroyer. Oh, and 3/4 of the systems are still at best have been drawn on a napkin and most of the rest came from a salvage yard. Oh and if you complete your next mission you may get money you can spend on some of the systems but if you do get that money you would be expected to do a lot more runs, quicker.
- TeMPOraL 7y agoA modern startup is a destroyer that had all its core operations systems fedexed to the yard, as they were rented from the cheapest third-party service providers, and nobody on the ship has ever bothered to read the contracts, much less know what a "SLA" is. Inevitably, at some point in the middle of the ocean, one of these systems will get remotely bricked because the company that provided it went under or got acquired.
- collyw 7y agoYour startup is probably just a variation of CRUD.
- 7y ago
- emerongi 7y agoThe author is not arguing for simplicity per se, but about being able to insert a human into a system that is normally automated. The example about the ship's steering system is perfect, actually. The system is not "simple" (if I were on a ship and the steering failed, I would be clueless), but it provides plenty of interjection points where a knowledgeable human can step in and either debug or fix the issue. It's those points that should be relatively simple - the whole system itself can be as complex as needed. Your system should be testable, debuggable and it should allow a human to step in and take control if needed. An open system is generally better than a closed one.
- taneq 7y agoAgreed, the first bit of the article isn't about simplicity as much as it is about visibility and the ability to directly control each aspect. This is very much in line with how modern SCADA systems are designed (and indeed the ship example is one of these). Each component can run automatically based on sensors and the state of its peers, or it can be directly controlled by the operator if required. The latter part of the article talks about simplicity but doesn't seem tightly tied to the earlier part.
- gk1 7y agoI considered using other labels besides "simple," like "redundant" or "straightforward," but decided no matter what word I used some ambiguity would remain. So I stick with "simple" and hoped the context would help carry the point across. I'm glad that it did, as it seems.
- tialaramex 7y agoHumans in the loop are likely to be a source of trouble, and you will need to measure to figure out if they provide any overall benefit other than giving you somebody to blame when that trouble happens. You need to ensure that the "human in the loop" scenario is obviously worse than automation or else humans, always over-confident of their abilities, will put themselves in the loop when it'd be far safer not. A good example of how to do this would be London's Docklands Light Railway. A DLR train is capable of autonomously going from one station to another. Trained operators are aboard every DLR train, and they have a key to a panel in the front of the train which reveals simple controls for manual operation. But very deliberately the maximum speed of the train with an operator at the controls is significantly lower than maximum speed under automation (there are two modes in fact, driving with the train still enforcing rules about where it can safely go which is a bit slower, and driving entirely without automation helping which is much slower). This underscores that manually driving the train is the way to sidestep a temporary problem, not a good idea in itself. The DLR has had several incidents in which trains collided, it will come as no surprise that they involved manually controlled trains. Humans are very flexible but mostly worse as part of your safety system, don't use humans if you can help it.
- deleted 7y ago[deleted]
- obscura 7y agoThere's way too much simplification and generalisation here, IMHO. "Choose tools that are simple to operate over those that promise the most features." Doing this very much depends on your requirements, doesn't it? What good is a simple-to-operate tool if it doesn't do what you need it to do? Sure, maybe you can simplify your requirements; then again, maybe not. The example of troubleshooting a whitepaper form very much depends on the design of the system. Maybe there's a reason for having multiple forms. If they all shared common architecture and the problem lies in what they share, it's not necessarily more difficult to troubleshoot many as opposed to just one. Moving from Marketo to HubSpot is good and well - for now. What about years from now? How did they end up with such complexity with the Marketo solution? Both solutions and requirements evolve. Down the line, you could end up with the same difficulties with HubSpot. I think it depends in part on how the organisation handles change. Lastly, I agree with the sentiment of Mateusz Górski, who commented on the page: is there any hard proof beyond anecdote to back up the article? If you expand the article's concept of a system to beyond just software, all you need is a hardware fault in a third-party hosting company with lousy support for your simple piece of software to be down for weeks.
- couchand 7y ago> all you need is a hardware fault in a third-party hosting company with lousy support for your simple piece of software to be down for weeks. I suspect the author would reply that you can just replace the hosting provider. Since the interface and division of responsibilities is clear, one host is completely replaceable with another.
- Papric0re 7y agoI share the idea of the author, but I miss actual talk about how we can make things better. Because his arguments are also true when it comes to security. What are alternatives to commonly used complex systems, like Docker/Kubernetes, Active Directory, Remote Access or just shared folder structures? How to keep the functionality of centralized management while making it simple? Would be happy to hear your thoughts on this.
- _wldu 7y agoThis tactic is also used to criticize/de-legitimize existing, working solutions when a new manager wants his old, familiar environment. "What Bob's team built is far too complex. If we replace it with the system I used at my last company, it will be much simpler." If I had a dollar for every time I've seen this type of thing, I'd be rich. So be careful you are not just playing corporate politics and changing solutions every time a new CXO joins.
- gk1 7y agoTotally. Funny enough, marketing platforms like those mentioned in the essay are usually the first things to get replaced when a new VP or CMO joins. Too often, unfortunately, that decision is made not because it's the best tool for the company but because it's the most familiar one.
- papito 7y agoInstagram was what, 12 employees when they got sold for a gazillion dollars? They all could fit into a van. Because they kept their system simple. It was (and still is) a monolith. Now imagine that they decided to go the microservices way. Multiply that team size by 10 at least. Don't solve problems you don't have.
- odiroot 7y agoI would really love to read their codebase.
- pmarreck 7y agoWhat’s their stack?
- grouchoboy 7y agoPrimary I think is a Django project https://www.youtube.com/watch?v=lx5WQjXLlq8&t=10s https://www.youtube.com/watch?v=lx5WQjXLlq8&t=10s
- pmarreck 7y agoI am straining to hide my lack of being impressed
- papito 7y agoRead all about it: https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c https://instagram-engineering.com/static-analysis-at-scale-a...
- justaguyhere 7y agotypically deploying to production around a hundred times per day That is insane! We make a release once a week at work and things still go wrong sometimes. I am in awe how they are able to pull this off, especially at their scale.
- numlock86 7y ago> Modifications before additions. > When new requirements come up, the tendency is to add layers on top of the existing system—by way of additional steps or integrations. Instead, see if the system’s core can be modified to meet the new requirements. I like the mention of this as it is - at least to me - really counter-intuitive.
- Koshkin 7y agoSurprised nobody has yet mentioned Einstein’s quote.
- geggam 7y agoThink this article is a different variation on the same theme. https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology
- csomar 7y agoThe author, probably, picked the worst example. Here is a pic: https://iro.nl/app/uploads/2018/12/P-67-onboard-the-BOKA-Vanguard-1000x750-c-default.jpg https://iro.nl/app/uploads/2018/12/P-67-onboard-the-BOKA-Van... No, the containership is not a simple system. It's very sophisticated one. It takes massive engineering effort (literally historical effort and knowledge) to build, massive resources for the material and outsourcing, very complex (like the pic) infrastructure in case of repair, satellites to guide the ship and check the weather, operators from all around the world to track the containers, massive ports to load-unload the shipments, etc... The operation of guiding the containership through the sea is a simple one but only once you have everything of the mentioned above in order. If you are building software (a full SaaS solution), you are not in the business of guiding the containership. You are in the business of building either the containership alone; or the whole related infrastructure (or parts of it). That's a huge undertaking that requires massive engineering efforts. It's not a simple task. If your task is a simple team that guide the container, you might make money for a while until everyone else figures out that you have no edge there and they can do your job for much less. The Instagram/Tinder examples are a bad one. It ignores the fact that to find the right (addictive/highly viral) app, you'll need (again) massive psychological knowledge and expertise into how primates brains work. We don't have that, so engineers are brute-forcing by making too many apps until one hits the jackpot.
- 32gbsd 7y agoPostulating about how things should be by non-experts has become a new e-sport. Then they try to apply it to software or startups as if its a apples to apples comparison.
- s3cur3 7y agoIn general, arguments by analogy tend to get hung up on the aptness of the analogy. I find it helps short-circuit a lot of long, unproductive discussions at work by just straightforwardly stating my thoughts—“we should do x because y,” rather than “system A is like system B (which does x because y), so we should do x.”
- 7y ago
- ddevault 7y agoThis article misses one crucial point in suggesting Looker over "a patchwork of custom scripts and APIs". The complexity doesn't go away because you offload it to Looker. When Looker goes down, your application is still in trouble. It's still your responsibility when your system goes down, even if you can point the finger at some third party service. Your system is still down! That being said, don't use Looker in the first place. Do not spy on people.
- gk1 7y agoSeveral people pointed out the fault in using Looker as an example. I very much welcome suggestions for a different product to plug there!
- thinkloop 7y agoReminds me a bit of home automation (which I enjoy). You manage a plethora of battery powered devices that need constant connection to finiky wifi with continuous access to your Linux server hub so that push and pull of data remains in tact so that you can have AI actuate a function through a "remote" - all to replace a light switch. Sometimes I wonder where this will end up. Might need 50 years before reliability and simplicity are up to par.
- pirate_dev 7y agoduh
- wonderwonder 7y agoI wrote my first deployed side project 5 years ago. It has been with the customer for the whole time and worked flawlessly while doing some fairly complex things, managing multiple processes for the customer. When I wrote it I had never done any web development and had not coded for 4+ years. C++ before that. Due to my lack of web development knowledge and massive time crunch, I built it with no framework, zero best practices and the most basic code I could build. The code is atrocious, and would not scale at all but it does what it is supposed to do and works because I did everything in the most basic way, there are no gotchas. Straight html, basic javascript and php on the backend. It interacts with google docs as well. Its almost to dumb to break. Sometimes we really do complicate things with all of our new fancy frameworks, microservices, states, etc. With that said, it would be impossible for anyone else to maintain and if anyone posted the code on the web I would probably be unemployed. But it works, very difficult to insert new features though.
- busterarm 7y agoI built a static jekyll site using wordpress as a backend for the content editors (jekyll build reads directly from the database) hosted on a handful of AWS VMs running NGINX and loadbalanced by HAProxy. The build system is automated in CircleCI. WordPress is hosted privately on a hosted WordPress provider. There's an ELK stack + plugins for logs and alerting and saltstack to trigger some basic automated sysadmin tasks. I left 3 years ago and the entire infrastructure has run on autopilot with zero downtime. Content editors and web developers still add to the content constantly. There is some dynamic content on the site handled by JavaScript and a few light APIs that I built to support the site. There's over 5000 pages of content. This simple infrastructure powers a website with a low-5 digit alexa rank for a business bringing in about a billion dollars a year. The entire infrastructure bill comes in under $100/mo and they no longer need to employ any backend engineers. We negotiated a contract agreement for me to support the site if they need it. They haven't.
- rsync 7y agoI gave a talk on this subject at CU last year and have, one way or another, spent my entire professional life thinking about this topic. I agree wholeheartedly that simple systems have less downtime. I would like to add a line from the talk that I give: Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways. rsync.net storage arrays typically have multi-hundred day uptimes. But across our entire network, our actual aggregate uptime is unimpressive - perhaps 99.99% ? Our SLA dictates 99.95. But when they do fail they fail in very, very boring ways. There are usually zero decisions to make in response to a failure.
- sysashi 7y agosorry for offtopic. there a broken link that I was interested to read on. its on https://www.rsync.net/resources/howto/rsync.html https://www.rsync.net/resources/howto/rsync.html ctrl+f "rsync snapshots are detailed here" I think given the topic, someone is expected to figure it out on their own (remember linux cake?), but I feel a bit lazy after a work day.. sorry
- rsync 7y agoSorry - that is in purpose ... The definitive page for rsync snapshots has been, and always will be, here: http://www.mikerubel.org/computers/rsync_snapshots/ http://www.mikerubel.org/computers/rsync_snapshots/ ... we actually don't want people to do rsync snapshots anymore because ZFS snapshots are much more efficient and use up less of their rsync.net account. If you change one bit of a file, the rsync snapshot method will cost the entire size of that file, since it's a hard link and is either broken or not broken. But if you just do a "dumb" sync to us and change one bit in the file, your ZFS snapshot will take up just one more bit. So not only do you get a much simpler backup script - you can really just do a dumb sync to us and let us rotate your snapshots - but you get more efficient space usage for changed files.
- sysashi 7y agoThanks for the link and explanations, appreciate it! I've read that page (i linked) inattentively, re-read 3 times that bold text next to the broken link and still asked that question... :facepalm:
- harrisreynolds 7y agoThis is a very important idea. I like the shipping container metaphor. I have often used the idea of a 2x4 (as in lumber) in building houses. The humble 2x4 is such a simple product, but when put together with nails and screws becomes the cornerstone of a complex structure. It is simple and it just works for its intended purpose. I am working hard to apply this to the design of Webase [1] which is a #nocode platform that is inherently complex. But I really believe that if we get the design right it will support a large number of sophisticated use-cases. Thank you for sharing! [1] https://www.webase.com https://www.webase.com
- dang 7y agoYou're overdoing these links. That's no doubt why you're getting downvoted and flagged. It's fine on HN to post your own work (1) in places where it's relevant, as long as (2) you only do it occasionally and (3) are also participating in the community in the intended ways, submitting interesting articles and having curious conversation. But when users break these rules of thumb (and you've been breaking all three of them!) it crosses into spamming. The community is very aware of that and really doesn't like it, so it's not in your interest. Doing this by hijacking top comments and threads about other people's projects is particularly not ok.
- harrisreynolds 7y agoThanks for the feedback. So far I have been getting positive feedback which is why I have continued to do this. I have been trying to add to the overall value of HN with insight in general and only including a link to my site if it is relevant. But I hear you and will adjust accordingly. I do sincerely appreciate the feedback as I have been steadily gaining karma and then out of no-where it when backwards today.
- dang 7y agoIt's a clear indication from the community that you're overdoing the promotional links. The fact that you haven't really been participating in any other way accentuates that impression. Readers check these things.
- dilap 7y agoI totally agree when designing man-made, synthetic systems. But it is interesting to consider that the most reliable, self-healing systems that exist -- animals! -- are incredibly complicated...
- rantwasp 7y agothere is a difference between complex and complicated. Also consider that systems where complexity serves a role are not complicated. Also: simple things that are aggregated into complex things through a hierarchy that is also simple can be reasoned about with failure happening subsystem level (and can be easily mitigated - think supervision trees in erlang)
- rantwasp 7y agoplug for my all time favorite paper in the subject https://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf https://web.mit.edu/2.75/resources/random/How%20Complex%20Sy... 5 pages of pure awesomeness
- awinter-py 7y ago> features don't justify complexity Such a good line. I've seen companies killed by hard-to-follow arguments advocating for technical complexity which serves feature goals which serve business goals. 'Good strategy is simple' from some strategy book is sometimes right; arguments that hold business goals hostage to technical complexity are trouble.
- matchagaucho 7y agoDeclarative coding tools are to blame here. This is a well written article, but the metaphor is a little stretched. Marketo and Salesforce are the "ship builders". Marketers are the crew operating the ship. The declarative tools in enterprise apps provide Marketers with conditional branch and execution capabilities without code structure, so the average Marketer just keeps adding layers of complexity. Whereas a Developer would continually refactor and make old code obsolete. The crew operating actual ships do not have anywhere near this level of access to the ships configuration, hence more reliability in the system.
- pm2222 7y ago`Uptime` of various *x systems comes to mind. Quality of code also matters.
- downerending 7y agoAs a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds like a good idea, but the extra complexity kills it. One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and would trip and then immediately fail, dropping all supplied power in an instant. The system was dramatically more reliable with this unit simply disabled.
- petschge 7y agoI said the same thing about UPS causing more trouble than preventing it. Then I moved to Africa and quickly learned that a local UPS can be VERY helpful.
- jniedrauer 7y agoThis has absolutely not been my experience, at least for well designed cloud infrastructure. High availability doesn't have to be complex or hard to understand. In most cases, automatic failovers and restarts on well designed architecture fix problems with zero end-user impact. It happens so seamlessly that my standard procedure these days is to fail over infrastructure if I want to restart it.
- downerending 7y agoWhat is this "well-designed infrastructure" of which you speak? ;-)
- shoo 7y ago> almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). I've seen similar things with rabbitmq, in situation where the load & number of events to process is tiny but HA is some checkbox item to deliver without a serious QA process to measure if the setup is actually working effectively. Number of events in production we genuinely needed multiple rabbitmq nodes in the cluster to cope with failure of one or more nodes: 0 . Number of times when rabbitmq got too excited processing large messages that it delayed heart beating & subsequently decided that there was a network partition when there was in fact no such thing, leading the prod support team to step in and manually recover the cluster: > 0
- deckarep 7y agoMy microservice that returns “Hello World” via HTTP has been running solid for 5+ years now with zero downtime.
- hinkley 7y agoSimple systems also benefit from Amdahl's Law, but with humans instead of CPUs. This is an observation I've been trying to sell others on for years and having only modest luck. When I can show them, they get it, but you have to push and push to get people there if words fail. When the shit hits the fan, there's an 80% chance that your most senior members will find the problem. They may find it quicker in the simple system, even, and others might get there first. But that last 20% is a very long tail, and even longer in a complex system. In a complex system, many of your members cannot participate in the triage process, because they don't know enough to know what's relevant. They slow down the people trying to work the problem trying to learn new things (good) or offering low-probability scenarios (bad). There is no 'All Hands on Deck' scenario for the complex system. To be responsive, you have to kick some or even a bunch of people out of the room, and once they leave they can't really participate. The first phase of a triage is getting a tight repro case. Many avenues are blocked until that happens. And some people have a knack for repro cases but aren't so great at debugging. With more people you get a higher quality repro, which cuts a lot of time off the rest of the process. Part of debugging is the cost of the verification versus the likelihood. Simple Systems afford the opportunity for people to test out unlikely but plausible scenarios that are in the long tail, without distracting from the more 'boring' checks already being done. And when bugs are identified prior to deployment, a simple system means you can hand the repro case to the responsible party and expect/demand that they get their fix on the first try.
- throw0101a 7y agoSee Gall's Law: > A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[9] * https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
- klysm 7y agoI find it mildly annoying when people state such nuggets of knowledge as “laws”. Sure, it’s generally true that you can’t go for the most sophisticated and powerful system at the first go, but sometimes the minimal working system is complex no? Anyway, it’s certainly not a “law” of the universe.
- stygiansonic 7y agoIt’s a colloquialism, like Moore’s law.
- sev0 7y ago"Law" can also be used for pithy statements that don't have to be particularly rigorous. C.f. Murphy's Law.
- K0SM0S 7y agoThis statement generalizes quite well all around the world to the other kind of law, that's voted in parliament/congress... haha! ;-) More seriously, I think the word OP is looking for is theory. Nuggets aren't theory, they're really empirical by essence — observations.
- mdonahoe 7y agoKlysm's Law: Nuggets of knowledge are not laws. Referring to them as such annoys Klysm.
- Smoosh 7y ago
- je42 7y agoI guess the main question to be asked is should be: is the complexity appropriate for the goal (i.e. Everything should be made as simple as possible, but no simpler. ) Hence, often a required step to fix complexity is to change the goal, in such a way that it can be implemented with less complexity.
- einpoklum 7y ago> Simple systems may have less downtime * Yeah, but often they do less / are less effective during uptime. * "simple" is an inspecific concept. Is the implementation simple? The API? The hardware? The requirements? * "simple" is a vague concept. Is a 1000-line C program more complex than a one-liner script in a fancy higher-level scripting language? Yes, if you ignore the 1,000,000 line abstract virtual machine, JIT compiler, interpreter, large standard library and what-not. Otherwise - maybe so, maybe no. > Modifications before additions I actually agree, but: * Managers don't like it. * Sometime this downtime happens before there's ever been any uptime... * Tomorrow you need something else, and the changes for that require re-engineering the parts you already re-engineered to best accommodate the previous addition.
- gdm85 7y ago> In the end, the system I put in place had 97% fewer processes (from 629 to 20) while providing all the same capabilities. I am suspicious of this statement. Is the author now supporting the company and defacto replacing the guy that left?
- aschatten 7y agoEverything seems logical and simple. But the examples lack details and realism. Take for example this quote: > In the end, the system I put in place had 97% fewer processes (from 629 to 20) while providing all the same capabilities. A bug that was found a few days later got resolved in four minutes. It‘s called refactoring. It‘s a constant process, that none likes to allocate time to, because it does not move metric immediately. Especially in marketing with a very fast changing landscape you go from 20 to 600+ very fast.