19 ms·

Systems design explains the world: volume 1

337 points by peapicker 6y ago

peapicker 6y ago
Apenwarr shares thoughts on system design, career paths, products, and innovation versus disruption.
Geminidog 6y ago
The problem with "systems design" is the lack of formality and the excessive use of conjecture. Nobody has a problem with Boxes and Arrows if those boxes and arrows have formal definitions (See category theory). The problem arises when those boxes and arrows represent vague unproven concepts with fuzzy meanings. For something to be more legitimate it needs to be formalized into a fixed set of axioms and formal logical rules. Relativity is an abstract concept but the rules it is based on are formalized. When nothing is formalized then nothing can be formally proven then the whole field descends into "design arguments" where examples and philosophical conjectures are thrown all over the place with no one able to really prove why one design pattern is better than another design pattern. The problem with this article is he never goes into these formal definitions of what a system is nor does he derive theorems. He just goes into example after example. His "revolutionary" thought here is that systems design may point to some underlying theory of design that applies to the world outside of computers. Big deal. We've seen these concepts many times but in different forms. "Design patterns," "Architecture" "System Architecture" and the industry just travels in circles around all of these concepts repeating the same old bs time after time never converging on definitive answers. The lack of formalization prevents us from truly knowing what an optimized design is so we just guess. Let me put it to you this way. Anytime you see the word "Design" it refers to a field where we have little knowledge of how things truly work. Nobody "designs" the shortest distance from point A to B. That value is calculated through formal theory. However we can't fully understand the best way for a "human" to get from point A, in California to point B in Colorado. We're simply not advanced enough to come up with a formal theory that can optimize the individuals preferences, time, space, energy, speed and comfort. Thus we "design" redundant solutions and hope that with each iteration in "design" we converge closer and closer to an optimal solution. But there's no way we can ever formally know if one solution is "better" than another. Design is another word for random intuitive guess. It is only deployed for problems where an optimized solution is unknown. "Design" represents an intense lack of knowledge. Anytime you see the word "design" or "architecture" you need to realize that you're descending into a field or talking to an "expert" where nobody knows anything really. As a result, the entire field of design travels in circles. Most new programming languages or frameworks are a useless attempt at producing an optimum solution but ends up being just another iteration of the endless circle. Is golang the optimum programming language? can we ever truly know if it's better? Are we truly pushing the field forward? I would say that for languages like typescript where the error surface area is provably smaller then JavaScript or Rust which also has a smaller error surface area then C++ that we have moved forward at getting closer to an optimum solution, but overall for things where a formal theory does not apply (type theory applies in the previous two examples) we aren't really converging on anything. The field is rife with "fake" formalizations and fake legitimacy. If someone has the job title "System architect" or "System Designer" you know it's complete BS. The guy is just making stuff up from intuition and experience, there is no formal science here. He is much closer to an "artist" then he is to a "mathematician", physicist" or "scientist" and should not command the same respect as the latter titles. The lack of formal legitimacy leaves a lot of room for BS. A good number of people achieve these roles simply by playing politics because there's really no way of knowing if the "architectures" they propose are inherently correct or "better" The endless circle is very apparent on HN where you just endlessly see new "Design" blog posts on the latest metaphor that can be used for "design." In this case the "design" is "systems design" and the metaphor is the "real world." Why do people endlessly write articles on new design metaphors? Because "design" indicates a lack of so much knowledge that it's really all they can do... make comparisons and metaphors rather then calculate actual optimum solutions.
- jacques_chester 6y ago
  If you prefer formalism, the cybernetics literature should appeal. I prefer the Systems Dynamics approach, because it's formal enough to avoid woo, but informal enough to avoid avoidance. But it's a mistake to create the general conclusion, from this blogpost, that a program aiming to describe, catalogue and comprehend phenomena across many fields in common terms is a fool's errand. It's been an ongoing program of research for most of a century now.
  - Geminidog 6y ago
    >But it's a mistake to create the general conclusion, from this blogpost, that a program aiming to describe, catalogue and comprehend phenomena across many fields in common terms is a fool's errand. It's been an ongoing program of research for most of a century now. It's not a fools errand but the way this article and many others approach this task is a fools errand. If he thinks colloquial use of "systems design" applies to various fields outside of computing, then he should formally define what he means by a "system" then prove his point. We don't need another blog post about some "design" metaphor.
    - jacques_chester 6y ago
      I had a slightly different complaint from yours, which is that the author doesn't seem conversant with the kinda-sorta-formal fields of study that already exist. Take for example the "chicken-egg" problem. Economists study this under "multi-sided markets" (which the author touches on, but late in the example), as part of the study of path dependency. Systems dynamics researchers have framed the economics work in their own terms of stocks and flows, but the essential structure is the same and can be reduced to equations. It's mostly calculus, sometimes there's some linear algebra. The best book in my view is still Sterman's Business Dynamics, which is much broader in scope than the title suggests (it's a play on earlier book titles). There will hopefully be a second edition in the next year or two. Edit: I should note however that when I see "system" I pattern match on what I know best. But there's a whole discipline of "System Engineering" which comes mostly from defence, aeronautical and astronautical fields. It has a high focus on formalisms to try to govern immensely complex technological efforts.
amelius 6y ago
> Joel Spolsky wrote Things you should never do, part 1 about the company-destroying effect of Netscape/Mozilla trying this. "They did it by making the single worst strategic mistake that any software company can make: they decided to rewrite the code from scratch." I'm not convinced. They only show a couple of examples, but there must also be counter-examples.
- nitrogen 6y ago
  I have one counterexample but it's a much smaller project: a bespoke ecommerce platform with somewhat tenuous integration into decades-old Oracle EBS backoffice systems was successfully replaced with a heavily customized open source ecommerce platform, with much better backoffice integration.
- l_t 6y ago
  I've been involved in rewrite-from-scratch projects that worked, because the scope of the system being rewritten was relatively small. So, I've concluded "rewrite from scratch" projects get a bad rap because they tend to also be very big, nebulously scoped projects, almost by definition. But it's totally unrelated to the fact that they're rewriting a codebase, and everything to do with the size and scope of the project to rewrite a codebase. Either way, though, we're lead to similar conclusions as the article, regarding refactoring. If a codebase/system is too large to rewrite in a single project, you have to break the work into incremental steps. Theoretically the only difference between "refactor" and "rewrite" is the scope of the affected code.
  - hliyan 6y ago
    Most writes we do as engineers are rewrites. The only difference is whether we're rewriting a statement, a function, a module or an entire system. You are correct. It is the size of the rewrite that matters.
- Twisol 6y ago
  To put it pithily, when "the problem has moved on from the solution", a rewrite might make sense. What I mean is that the existing software has existed for long enough, without incremental upkeep and rework, that the problem it was originally solving has evolved to the point that the existing software is not only inadequate, it's wholly inappropriate. I'm part of one of those rewrites right now, and while it's true that it's taken much longer and had other ancillary issues, it's clear that the problem really has moved on, and there are needs that can't be addressed without a fundamentally different solution. It's kind of trading a hard problem for a harder one: we're forced to really understand the domain we're serving, rather than simply reimplement a set of existing capabilities. (But then, I don't think maintainable, evolvable software can reliably be created without understanding the domain to begin with.)
abc_lisper 6y ago
Very nice article. I always struggled with design, not because I didn’t know how to, but the feeling of uneasiness that comes up when we don’t have the details or imagination to see the repercussions of our designs. I’m stuck as a senior engineer, even though I have designed and implemented systems, because I think the veneer of design knowledge when talking to others is pretty thin, and as someone who reasons by causality and simulation, it’s hard to hide it. I always wonder how people get around this. What books did you read, general ones, not the ones you use for interviews that crow “here’s how to design twitter”
- contingencies 6y ago
  There is no external source of truth for complex or wicked design problems (other than the intended use or market). The best you can hope for is a library of stimulating reference works, exploratory processes, personal habits and muses to assist. It has been said with some truth that successful design tends to iterate between broader and highly specific contexts, and utilizes an iterative approach to solutions rather than attempting to successfully anticipate all requirements with one initial design. The author of this piece seems to be using "systems design" as a synonym for the broader perspective design thinking, both at the level of emergent system properties and commercial implications, and while he raises some good points it seems a bit shotgun in focus and the long-form article format a mismatch for the subject matter. I have a library of largely design-related fortunes which I start my terminal with, they are useful to me for design thinking stimulation purposes and are optimized for brevity. Many are derived from books or papers. https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
jiveturkey 6y ago
the big tech company he refers to is google
deleted 6y ago
[deleted]
NotPavlovsDog 6y ago
The statement "Which came first, HTML5 web browsers or HTML5 web content? Neither, of course. They evolved in loose synchronization, tracing from way before HTML itself, growing slowly and then quickly in popularity along the way." Prompted me to abandon reading the rest of the article. There were committees and centralized coordination efforts involved in HTML5, just like the languages before it, for the standard and for the participating vendor own implementations. Development of the standard came first. These sweeping generalizations and grand conclusions make an impression that the example was pulled along the ears to serve some grand statement. It is the statement of the article that appears to be loose.
- escape_goat 6y ago
  You abandoned the article because you disputed the characterization of HTML5 development in a throw-away example that had nothing to do with the actual topic of the article just about exactly midway through it? Get out of here. You were already annoyed by the article's tendency to use the sweeping and made-to-purpose generalizations that you feel that this represents.
- skybrian 6y ago
  This seems like an unsympathetic reading. The point seems to be that standards usually aren't simply invented out of nothing; they take existing practice as input. Sure, there was a standards process, but the people involved looked into what browsers did and how popular particular HTML constructs were in the millions of websites that already existed when the HTML5 standard was being written, and tried not to break them. The input to their design process came from many sources. Before HTML5 formally existed, there were many web pages that more-or-less did what the standard said they should do, and browsers that more-or-less worked with those web pages.
  - mprovost 6y ago
    Exactly. For the IETF, defining a draft standard (usually) requires two independent implementations first. Efforts to define standards before trying to implement them are usually disasters.
abanayev 6y ago
I’m shocked by how aggressive a lot of the comments in this thread have been. This is a well written article, regardless of where anyone stands on questions like whether to rewrite vs improve, build vs buy, etc. Questions of systems design are anyway not answerable by appeal to a formal theory, at least not yet, so like most gray questions of life, they are not ones where one should take a fixed stand. There are too many unpredictable or emergent effects of an answer to such questions.
- Geminidog 6y ago
  There's only one comment that addresses your point. The rest of the comments actually agree with the article. Either way I'm more arguing against the plethora of endless informal exposes into design and design metaphors that I see on HN everyday. These things do little to move the field forward no matter how eloquently they're written. The endless flat circle continues to rotate. It's like an overdone hollywood genre.
raldi 6y ago
If you think Systems Design is bullshit, you haven't played enough Factorio.
- TameAntelope 6y ago
  I prefer to watch Nilaus do it on YouTube/Twitch, because his job prior to becomming a full-time YouTube content creator was leading an engineering organization, and got his degree in "Transport and Logistics". He understands the problems Factorio is presenting, and the "state of the art" for how to solve them, and it's a ton of fun to watch what he focuses on, what gets backburnered, etc. I for sure recommend his channel if you like to see Factorio as (IMO) it's supposed to be played.
tome 6y ago
The ARM vs Intel story of leaving the low margin behind to focus on the high margin reminds me of USA vs China: https://www.bloomberg.com/news/articles/2020-12-26/covid-fallout-means-china-to-overtake-u-s-economy-earlier https://www.bloomberg.com/news/articles/2020-12-26/covid-fal...
- hyko 6y ago
  The innovator’s dilemma doesn’t apply to USA vs China; it’s not like the USA would want to become a developing country again. Besides that, seven years is a long time: China obviously has favourable characteristics in terms of population size, but it also faces enormous challenges and there are no guarantees, e.g: https://www.chicagotribune.com/news/ct-xpm-1995-04-10-9504100029-story.html https://www.chicagotribune.com/news/ct-xpm-1995-04-10-950410...
raldi 6y ago
One of the greatest System Design stories of all time is the evolution of mammals. Warm blood? A huge, complicated heart? Are you insane? The metabolism rates will be off the charts. Record-breaking spin-up times before adulthood? Where the parent animals will have to actively care for their helpless young? And spend bodily resources to produce a fluid with which to constantly feed each one? How could they ever compete with a species that lays 10,000 eggs a year, dashing off after — or before — they hatch? And yet, as They Might Be Giants put it, these disruptive designs, so seemingly flawed, made the difference between extinction in the cold and explosive, radiating growth. P.S. Did you recognize the chicken-and-egg problem in the second paragraph?
- systemvoltage 6y ago
  To be fair, the last statement about extinction in the cold and explosive growth is largely due to the size of the brain. Humans figured out techniques to survive in the cold even while being warm blooded. No thick fur is necessary when we can just kill an animal and use its skin to get warmth. Then we discovered fire.
  - raldi 6y ago
    I'm talking about a disruption story that long predated humans: https://genius.com/7391311 https://genius.com/7391311
- Geminidog 6y ago
  >And yet, as They Might Be Giants put it, these disruptive designs, so seemingly flawed, made the difference between extinction in the cold and explosive, radiating growth. These designs aren't seemingly flawed, they are flawed. Evolution produces flawed designs by "design". Evolution trends towards survival. One thing critical for survival is equilibrium of the ecosystem so on a macro level evolution will trend towards equilibrium even if that necessitates the need for flawed designs. In fact, flawed designs are critical to ecosystem survival. It works because competing designs to the flawed design are also just as flawed. An invasive species on an island ecosystem is a perfect example of an over efficient design destroying a local ecosystem and in turn eventually destroying itself. This is why ebola didn't spread as far as covid-19. Ebola is too efficient at killing thus reducing transmission rate and burning out the local ecosystem and thereby destroying itself before it got too far. The less efficient virus is the one that spread across the globe.
Pasorrijer 6y ago
I think one reason for the negative comments (speaking as someone trained as a Systems Engineer, with 10+ years of experience and now working as a Consultant and performing Systems Engineering occasionally) is that the majority of the "Engineerings" choose to quantify anything fuzzy in a certain way, and then state that it doesn't matter. For example, the human factor. Whereas the reason Systems Engineering is so abstract and high level is the assumption that those unquantifiable factors have to be accounted for in "boxes and arrows", regardless of the fact that they can't be quantitatively engineered.... And many can't handle the fact that these things can't be quantitatively engineered [Edit to add the statement post ellipses]
- hliyan 6y ago
  As a heavily 'left-brained' person with an engineering background, I've recently (in my middle age) started to appreciate those who pursue social sciences more. They were a group I used to look down on because of the lack of rigor in their work, but recently I have come to realize that the problem is that the atomic unit of their domain (a human mind) is orders of magnitude more complex than that atomic units of all my favorite disciplines (physics: forces, chemistry: actual atoms, programming: instructions). While people like myself dismiss the problem (read: run away from it), they at least possess the fortitude to attempt it, even while stumbling.
  - thrashh 6y ago
    The lack of rigor is pretty prevalent in life (legal systems, human communication, etc.) and it’s awesome because it adds a whole extra dimension of complexity that could not exist if things were rigorous. For example, I can test whether my friend is sad about something without explicitly asking them and thus triggering them. Or I try to see if someone is interested in me without explicitly asking and making it awkward for either of us. Or a judge can choose to be more lenient to someone trespassing to save their dog. It adds this huge “gray area” that can exist when things are wishy washy and it’s like the lubricant that smooths out unique situations that can’t possibly be covered in rigorous system.
    - 6y ago
alisonkisk 6y ago
In my experience, "systema design" has nothing to do with climbing the career ladder, except in a narrow sense of "architecture astronauts claiming credit for 'designing' the work their teammates actually designed and implemented. Most people get promoted by inventing useful or at least measurable new tech. building a horrible system doesn't hurt anyone, since text debt doesn't collapse the project until after promotion.
lambda 6y ago
One of the funny things about the Intel/Arm situation is that in many ways, it mirrors how Intel itself started out on low-end microcontrollers and microprocessors for PCs, and grew to displace pretty much everyone else for the high end supercomputers, data centers, and the like. For many years, it has been fairly apparent that the same thing was happening with Arm; when Arm thoroughly cemented its position on top of the smartphone market, it seemed pretty clear that they would eventually be able to push up into the laptop market and beyond, the main question was when. Unfortunately for Intel, they've been hit by a couple of major blows in quick succession; Spectre and other microarchitectural vulnerabilities, and the fact that the mitigations wiped out a lot of their architectural advantages, their trouble with manufacturing at new process nodes, allowing AMD to take the lead from them on high-performance x86, and Arm finally catching up with them in the desktop/laptop space, and also starting to make some small inroads in the server space (though mostly on cost, not performance at the moment). One thing which may have made them more susceptible to the innovators dilemma problem is the fact that they were already a victim of the second system effect with their Itanium failure. They poured a lot of resources into a major new ISA, and it was a flop and was replaced by the incremental x86-64. Despite that, their incremental improvements on the x86 architecture managed to wipe out all of the RISC competitors on the high end; so there's probably even more organizational oppositions to major new architectural changes which could have kept Arm at bay. I think it's interesting that it has taken this long for RISC to really pay off. The whole idea behind RISC was to optimize the ISA for modern, pipelined, OoO architectures, but it turns out that for a long time, a bit of extra complexity in the instruction decoder wasn't that much of a price to pay, and things like Moore's law and other architectural improvements kept it mostly irrelevant as a competitive advantage. But finally, Intel has run out of running room on process dominance, and other things have happened like hitting more power limits and software that's able to take advantage of more available threads and relaxed memory models. Once you start to try to run more threads, the extra cost and complexity of those decoders starts to add up, and the strong memory ordering constraints mean that your hands are a bit more tied on huge OoO dispatch. Anyhow, Intel is still the biggest semiconductor company in the world, and they have weathered other major issues like the famous Pentium floating point bug, Itanium, and Spectre. It will take a lot to dethrone them, and with their revenue and market share, they can lose a lot of battles and invest a lot of money into new technology or companies before they lose the war. But the last few years make it clear that they need some big wins soon, or they risk turning into the next IBM, slowly losing relevance as they milk their cash cows while the world moves on.
lincpa 6y ago
> "Systems design" is a branch of study that tries to find universal architectural patterns that are valid across disciplines. It's universal architectural patterns that are valid across disciplines. The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow

337 points by peapicker 6y ago

Apenwarr shares thoughts on system design, career paths, products, and innovation versus disruption.

The problem with "systems design" is the lack of formality and the excessive use of conjecture. Nobody has a problem with Boxes and Arrows if those boxes and arrows have formal definitions (See category theory). The problem arises when those boxes and arrows represent vague unproven concepts with fuzzy meanings. For something to be more legitimate it needs to be formalized into a fixed set of axioms and formal logical rules. Relativity is an abstract concept but the rules it is based on are formalized. When nothing is formalized then nothing can be formally proven then the whole field descends into "design arguments" where examples and philosophical conjectures are thrown all over the place with no one able to really prove why one design pattern is better than another design pattern. The problem with this article is he never goes into these formal definitions of what a system is nor does he derive theorems. He just goes into example after example. His "revolutionary" thought here is that systems design may point to some underlying theory of design that applies to the world outside of computers. Big deal. We've seen these concepts many times but in different forms. "Design patterns," "Architecture" "System Architecture" and the industry just travels in circles around all of these concepts repeating the same old bs time after time never converging on definitive answers. The lack of formalization prevents us from truly knowing what an optimized design is so we just guess. Let me put it to you this way. Anytime you see the word "Design" it refers to a field where we have little knowledge of how things truly work. Nobody "designs" the shortest distance from point A to B. That value is calculated through formal theory. However we can't fully understand the best way for a "human" to get from point A, in California to point B in Colorado. We're simply not advanced enough to come up with a formal theory that can optimize the individuals preferences, time, space, energy, speed and comfort. Thus we "design" redundant solutions and hope that with each iteration in "design" we converge closer and closer to an optimal solution. But there's no way we can ever formally know if one solution is "better" than another. Design is another word for random intuitive guess. It is only deployed for problems where an optimized solution is unknown. "Design" represents an intense lack of knowledge. Anytime you see the word "design" or "architecture" you need to realize that you're descending into a field or talking to an "expert" where nobody knows anything really. As a result, the entire field of design travels in circles. Most new programming languages or frameworks are a useless attempt at producing an optimum solution but ends up being just another iteration of the endless circle. Is golang the optimum programming language? can we ever truly know if it's better? Are we truly pushing the field forward? I would say that for languages like typescript where the error surface area is provably smaller then JavaScript or Rust which also has a smaller error surface area then C++ that we have moved forward at getting closer to an optimum solution, but overall for things where a formal theory does not apply (type theory applies in the previous two examples) we aren't really converging on anything. The field is rife with "fake" formalizations and fake legitimacy. If someone has the job title "System architect" or "System Designer" you know it's complete BS. The guy is just making stuff up from intuition and experience, there is no formal science here. He is much closer to an "artist" then he is to a "mathematician", physicist" or "scientist" and should not command the same respect as the latter titles. The lack of formal legitimacy leaves a lot of room for BS. A good number of people achieve these roles simply by playing politics because there's really no way of knowing if the "architectures" they propose are inherently correct or "better" The endless circle is very apparent on HN where you just endlessly see new "Design" blog posts on the latest metaphor that can be used for "design." In this case the "design" is "systems design" and the metaphor is the "real world." Why do people endlessly write articles on new design metaphors? Because "design" indicates a lack of so much knowledge that it's really all they can do... make comparisons and metaphors rather then calculate actual optimum solutions.

jacques_chester 6y ago
If you prefer formalism, the cybernetics literature should appeal. I prefer the Systems Dynamics approach, because it's formal enough to avoid woo, but informal enough to avoid avoidance. But it's a mistake to create the general conclusion, from this blogpost, that a program aiming to describe, catalogue and comprehend phenomena across many fields in common terms is a fool's errand. It's been an ongoing program of research for most of a century now.
- Geminidog 6y ago
  >But it's a mistake to create the general conclusion, from this blogpost, that a program aiming to describe, catalogue and comprehend phenomena across many fields in common terms is a fool's errand. It's been an ongoing program of research for most of a century now. It's not a fools errand but the way this article and many others approach this task is a fools errand. If he thinks colloquial use of "systems design" applies to various fields outside of computing, then he should formally define what he means by a "system" then prove his point. We don't need another blog post about some "design" metaphor.
  - jacques_chester 6y ago
    I had a slightly different complaint from yours, which is that the author doesn't seem conversant with the kinda-sorta-formal fields of study that already exist. Take for example the "chicken-egg" problem. Economists study this under "multi-sided markets" (which the author touches on, but late in the example), as part of the study of path dependency. Systems dynamics researchers have framed the economics work in their own terms of stocks and flows, but the essential structure is the same and can be reduced to equations. It's mostly calculus, sometimes there's some linear algebra. The best book in my view is still Sterman's Business Dynamics, which is much broader in scope than the title suggests (it's a play on earlier book titles). There will hopefully be a second edition in the next year or two. Edit: I should note however that when I see "system" I pattern match on what I know best. But there's a whole discipline of "System Engineering" which comes mostly from defence, aeronautical and astronautical fields. It has a high focus on formalisms to try to govern immensely complex technological efforts.

amelius 6y ago

> Joel Spolsky wrote Things you should never do, part 1 about the company-destroying effect of Netscape/Mozilla trying this. "They did it by making the single worst strategic mistake that any software company can make: they decided to rewrite the code from scratch." I'm not convinced. They only show a couple of examples, but there must also be counter-examples.

nitrogen 6y ago
I have one counterexample but it's a much smaller project: a bespoke ecommerce platform with somewhat tenuous integration into decades-old Oracle EBS backoffice systems was successfully replaced with a heavily customized open source ecommerce platform, with much better backoffice integration.
l_t 6y ago
I've been involved in rewrite-from-scratch projects that worked, because the scope of the system being rewritten was relatively small. So, I've concluded "rewrite from scratch" projects get a bad rap because they tend to also be very big, nebulously scoped projects, almost by definition. But it's totally unrelated to the fact that they're rewriting a codebase, and everything to do with the size and scope of the project to rewrite a codebase. Either way, though, we're lead to similar conclusions as the article, regarding refactoring. If a codebase/system is too large to rewrite in a single project, you have to break the work into incremental steps. Theoretically the only difference between "refactor" and "rewrite" is the scope of the affected code.
- hliyan 6y ago
  Most writes we do as engineers are rewrites. The only difference is whether we're rewriting a statement, a function, a module or an entire system. You are correct. It is the size of the rewrite that matters.
Twisol 6y ago
To put it pithily, when "the problem has moved on from the solution", a rewrite might make sense. What I mean is that the existing software has existed for long enough, without incremental upkeep and rework, that the problem it was originally solving has evolved to the point that the existing software is not only inadequate, it's wholly inappropriate. I'm part of one of those rewrites right now, and while it's true that it's taken much longer and had other ancillary issues, it's clear that the problem really has moved on, and there are needs that can't be addressed without a fundamentally different solution. It's kind of trading a hard problem for a harder one: we're forced to really understand the domain we're serving, rather than simply reimplement a set of existing capabilities. (But then, I don't think maintainable, evolvable software can reliably be created without understanding the domain to begin with.)

abc_lisper 6y ago

Very nice article. I always struggled with design, not because I didn’t know how to, but the feeling of uneasiness that comes up when we don’t have the details or imagination to see the repercussions of our designs. I’m stuck as a senior engineer, even though I have designed and implemented systems, because I think the veneer of design knowledge when talking to others is pretty thin, and as someone who reasons by causality and simulation, it’s hard to hide it. I always wonder how people get around this. What books did you read, general ones, not the ones you use for interviews that crow “here’s how to design twitter”

contingencies 6y ago
There is no external source of truth for complex or wicked design problems (other than the intended use or market). The best you can hope for is a library of stimulating reference works, exploratory processes, personal habits and muses to assist. It has been said with some truth that successful design tends to iterate between broader and highly specific contexts, and utilizes an iterative approach to solutions rather than attempting to successfully anticipate all requirements with one initial design. The author of this piece seems to be using "systems design" as a synonym for the broader perspective design thinking, both at the level of emergent system properties and commercial implications, and while he raises some good points it seems a bit shotgun in focus and the long-form article format a mismatch for the subject matter. I have a library of largely design-related fortunes which I start my terminal with, they are useful to me for design thinking stimulation purposes and are optimized for brevity. Many are derived from books or papers. https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

jiveturkey 6y ago

the big tech company he refers to is google

deleted 6y ago

[deleted]

NotPavlovsDog 6y ago

The statement "Which came first, HTML5 web browsers or HTML5 web content? Neither, of course. They evolved in loose synchronization, tracing from way before HTML itself, growing slowly and then quickly in popularity along the way." Prompted me to abandon reading the rest of the article. There were committees and centralized coordination efforts involved in HTML5, just like the languages before it, for the standard and for the participating vendor own implementations. Development of the standard came first. These sweeping generalizations and grand conclusions make an impression that the example was pulled along the ears to serve some grand statement. It is the statement of the article that appears to be loose.

escape_goat 6y ago
You abandoned the article because you disputed the characterization of HTML5 development in a throw-away example that had nothing to do with the actual topic of the article just about exactly midway through it? Get out of here. You were already annoyed by the article's tendency to use the sweeping and made-to-purpose generalizations that you feel that this represents.
skybrian 6y ago
This seems like an unsympathetic reading. The point seems to be that standards usually aren't simply invented out of nothing; they take existing practice as input. Sure, there was a standards process, but the people involved looked into what browsers did and how popular particular HTML constructs were in the millions of websites that already existed when the HTML5 standard was being written, and tried not to break them. The input to their design process came from many sources. Before HTML5 formally existed, there were many web pages that more-or-less did what the standard said they should do, and browsers that more-or-less worked with those web pages.
- mprovost 6y ago
  Exactly. For the IETF, defining a draft standard (usually) requires two independent implementations first. Efforts to define standards before trying to implement them are usually disasters.

abanayev 6y ago

I’m shocked by how aggressive a lot of the comments in this thread have been. This is a well written article, regardless of where anyone stands on questions like whether to rewrite vs improve, build vs buy, etc. Questions of systems design are anyway not answerable by appeal to a formal theory, at least not yet, so like most gray questions of life, they are not ones where one should take a fixed stand. There are too many unpredictable or emergent effects of an answer to such questions.

Geminidog 6y ago
There's only one comment that addresses your point. The rest of the comments actually agree with the article. Either way I'm more arguing against the plethora of endless informal exposes into design and design metaphors that I see on HN everyday. These things do little to move the field forward no matter how eloquently they're written. The endless flat circle continues to rotate. It's like an overdone hollywood genre.

raldi 6y ago

If you think Systems Design is bullshit, you haven't played enough Factorio.

TameAntelope 6y ago
I prefer to watch Nilaus do it on YouTube/Twitch, because his job prior to becomming a full-time YouTube content creator was leading an engineering organization, and got his degree in "Transport and Logistics". He understands the problems Factorio is presenting, and the "state of the art" for how to solve them, and it's a ton of fun to watch what he focuses on, what gets backburnered, etc. I for sure recommend his channel if you like to see Factorio as (IMO) it's supposed to be played.

tome 6y ago

The ARM vs Intel story of leaving the low margin behind to focus on the high margin reminds me of USA vs China: https://www.bloomberg.com/news/articles/2020-12-26/covid-fallout-means-china-to-overtake-u-s-economy-earlier https://www.bloomberg.com/news/articles/2020-12-26/covid-fal...

hyko 6y ago
The innovator’s dilemma doesn’t apply to USA vs China; it’s not like the USA would want to become a developing country again. Besides that, seven years is a long time: China obviously has favourable characteristics in terms of population size, but it also faces enormous challenges and there are no guarantees, e.g: https://www.chicagotribune.com/news/ct-xpm-1995-04-10-9504100029-story.html https://www.chicagotribune.com/news/ct-xpm-1995-04-10-950410...

raldi 6y ago

One of the greatest System Design stories of all time is the evolution of mammals. Warm blood? A huge, complicated heart? Are you insane? The metabolism rates will be off the charts. Record-breaking spin-up times before adulthood? Where the parent animals will have to actively care for their helpless young? And spend bodily resources to produce a fluid with which to constantly feed each one? How could they ever compete with a species that lays 10,000 eggs a year, dashing off after — or before — they hatch? And yet, as They Might Be Giants put it, these disruptive designs, so seemingly flawed, made the difference between extinction in the cold and explosive, radiating growth. P.S. Did you recognize the chicken-and-egg problem in the second paragraph?

systemvoltage 6y ago
To be fair, the last statement about extinction in the cold and explosive growth is largely due to the size of the brain. Humans figured out techniques to survive in the cold even while being warm blooded. No thick fur is necessary when we can just kill an animal and use its skin to get warmth. Then we discovered fire.
- raldi 6y ago
  I'm talking about a disruption story that long predated humans: https://genius.com/7391311 https://genius.com/7391311
Geminidog 6y ago
>And yet, as They Might Be Giants put it, these disruptive designs, so seemingly flawed, made the difference between extinction in the cold and explosive, radiating growth. These designs aren't seemingly flawed, they are flawed. Evolution produces flawed designs by "design". Evolution trends towards survival. One thing critical for survival is equilibrium of the ecosystem so on a macro level evolution will trend towards equilibrium even if that necessitates the need for flawed designs. In fact, flawed designs are critical to ecosystem survival. It works because competing designs to the flawed design are also just as flawed. An invasive species on an island ecosystem is a perfect example of an over efficient design destroying a local ecosystem and in turn eventually destroying itself. This is why ebola didn't spread as far as covid-19. Ebola is too efficient at killing thus reducing transmission rate and burning out the local ecosystem and thereby destroying itself before it got too far. The less efficient virus is the one that spread across the globe.

Pasorrijer 6y ago

I think one reason for the negative comments (speaking as someone trained as a Systems Engineer, with 10+ years of experience and now working as a Consultant and performing Systems Engineering occasionally) is that the majority of the "Engineerings" choose to quantify anything fuzzy in a certain way, and then state that it doesn't matter. For example, the human factor. Whereas the reason Systems Engineering is so abstract and high level is the assumption that those unquantifiable factors have to be accounted for in "boxes and arrows", regardless of the fact that they can't be quantitatively engineered.... And many can't handle the fact that these things can't be quantitatively engineered [Edit to add the statement post ellipses]

hliyan 6y ago
As a heavily 'left-brained' person with an engineering background, I've recently (in my middle age) started to appreciate those who pursue social sciences more. They were a group I used to look down on because of the lack of rigor in their work, but recently I have come to realize that the problem is that the atomic unit of their domain (a human mind) is orders of magnitude more complex than that atomic units of all my favorite disciplines (physics: forces, chemistry: actual atoms, programming: instructions). While people like myself dismiss the problem (read: run away from it), they at least possess the fortitude to attempt it, even while stumbling.
- thrashh 6y ago
  The lack of rigor is pretty prevalent in life (legal systems, human communication, etc.) and it’s awesome because it adds a whole extra dimension of complexity that could not exist if things were rigorous. For example, I can test whether my friend is sad about something without explicitly asking them and thus triggering them. Or I try to see if someone is interested in me without explicitly asking and making it awkward for either of us. Or a judge can choose to be more lenient to someone trespassing to save their dog. It adds this huge “gray area” that can exist when things are wishy washy and it’s like the lubricant that smooths out unique situations that can’t possibly be covered in rigorous system.
  - 6y ago

alisonkisk 6y ago

In my experience, "systema design" has nothing to do with climbing the career ladder, except in a narrow sense of "architecture astronauts claiming credit for 'designing' the work their teammates actually designed and implemented. Most people get promoted by inventing useful or at least measurable new tech. building a horrible system doesn't hurt anyone, since text debt doesn't collapse the project until after promotion.

lambda 6y ago

One of the funny things about the Intel/Arm situation is that in many ways, it mirrors how Intel itself started out on low-end microcontrollers and microprocessors for PCs, and grew to displace pretty much everyone else for the high end supercomputers, data centers, and the like. For many years, it has been fairly apparent that the same thing was happening with Arm; when Arm thoroughly cemented its position on top of the smartphone market, it seemed pretty clear that they would eventually be able to push up into the laptop market and beyond, the main question was when. Unfortunately for Intel, they've been hit by a couple of major blows in quick succession; Spectre and other microarchitectural vulnerabilities, and the fact that the mitigations wiped out a lot of their architectural advantages, their trouble with manufacturing at new process nodes, allowing AMD to take the lead from them on high-performance x86, and Arm finally catching up with them in the desktop/laptop space, and also starting to make some small inroads in the server space (though mostly on cost, not performance at the moment). One thing which may have made them more susceptible to the innovators dilemma problem is the fact that they were already a victim of the second system effect with their Itanium failure. They poured a lot of resources into a major new ISA, and it was a flop and was replaced by the incremental x86-64. Despite that, their incremental improvements on the x86 architecture managed to wipe out all of the RISC competitors on the high end; so there's probably even more organizational oppositions to major new architectural changes which could have kept Arm at bay. I think it's interesting that it has taken this long for RISC to really pay off. The whole idea behind RISC was to optimize the ISA for modern, pipelined, OoO architectures, but it turns out that for a long time, a bit of extra complexity in the instruction decoder wasn't that much of a price to pay, and things like Moore's law and other architectural improvements kept it mostly irrelevant as a competitive advantage. But finally, Intel has run out of running room on process dominance, and other things have happened like hitting more power limits and software that's able to take advantage of more available threads and relaxed memory models. Once you start to try to run more threads, the extra cost and complexity of those decoders starts to add up, and the strong memory ordering constraints mean that your hands are a bit more tied on huge OoO dispatch. Anyhow, Intel is still the biggest semiconductor company in the world, and they have weathered other major issues like the famous Pentium floating point bug, Itanium, and Spectre. It will take a lot to dethrone them, and with their revenue and market share, they can lose a lot of battles and invest a lot of money into new technology or companies before they lose the war. But the last few years make it clear that they need some big wins soon, or they risk turning into the next IBM, slowly losing relevance as they milk their cash cows while the world moves on.

lincpa 6y ago

> "Systems design" is a branch of study that tries to find universal architectural patterns that are valid across disciplines. It's universal architectural patterns that are valid across disciplines. The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow