16 ms·

Against Essential and Accidental Complexity

246 points by weinzierl 6y ago

giovannibonetti 6y ago
It is easy to see that the total complexity has been severely reduced in the last years when you look at how many small companies are doing things that required an army of people a few decades ago. For example, running the technology for a real-time banking system nowadays can be done by perhaps even a single developer (with enough experience) thanks to open source, cloud computing and advances in software and hardware. Now we even have distributed relational databases that are relatively straightforward to operate, like CockroachDB.
- pron 6y ago
  First, Brooks wasn't talking about that. He was talking about productivity boosts due to programming language design and programming methodology. More specifically, he did not exclude improvements to both kinds of complexity overall, but only those that drastically change their relative proportion. Second, in the early 90s I had a summer job at a company whose old ERP, which had been written in a language called Business Basic and ran on a minicomputer, was maintained and operated by a single person.
  - jasode 6y ago
    >, Brooks wasn't talking about that. He was talking about productivity boosts due to programming language design and programming methodology. Brooks was also talking about non-programming language advancements as a possible "silver bullets". See the pdf and look for the following sections that are not about programming syntax: Hopes for the Silver Artificial Intelligence Environments and Tools Promising Attacks on the Conceptual Essence Buy versus Build He underestimated AI machine learning. With hindsight, we see that the deep-layer neural net combined with GPU beat human rule-based programming for automatic language translations, DeepMind AlphaZero beats expert programmers hand-tweaking IBM DeepBlue, etc. EDIT reply to : ", nor have they reduced the overall cost of programming by 10x even after more than three decades." I'm not convinced of that. The issue with your conclusion is that it omits the increasing expectations of more complex systems. I'd argue we did get 10x improvement -- if -- we hold the size & complexity of the system constant. E.g. write 1970s style text-based code to summarize sales revenue by region and output it to green bar dot matrix printers. This was tedious work in old COBOL but much easier today with Python Pandas or even no code tools like MS Excel. The invisible factor we often overlook is that our demand for software to do more complex things will always outpace the smaller productivity increases of the tools. This makes it look like our newer programming tools never gave us 10x improvement when it actually did.
    - pron 6y ago
      How did he underestimate statistical machine learning? Whatever achievements were made, they did not take place within a decade, nor have they reduced the overall cost of programming by 10x even after more than three decades. And, indeed, the one thing that Brooks presented as being the most promising direction, i.e. buy vs. build. He said that unlike changes to programming languages and methodology that wouldn't give a huge boost, buy vs. build might (although he wasn't sure about that). So he was exactly right about that. Also, 1986 wasn't 1966. In 1986 people didn't write simple software in COBOL. They used things like Magic and Business Basic and even Smalltalk, and, shortly after, Visual Basic and Access. Excel was released in 1987, and we had spreadsheets in the early 80s, too (and Brooks explicitly mentions them in No Silver Bullet). RAD tools and "no code" was very much in vogue in the late 80s and early 90s. That was Brooks's present, not future. He even says that this is the right direction: I believe the single most powerful software-productivity strategy for many organizations today is to equip the computer-naive intellectual workers who are on the firing line with personal computers and good generalized writing, drawing, file, and spreadsheet programs and then to turn them loose. The same strategy, carried out with generalized mathematical and statistical packages and some simple programming capabilities, will also work for hundreds of laboratory scientists. When generalised, Brooks's prediction amounts to expecting diminishing returns due to reduction of accidental complexity, and we're seeing exactly that. > I'd argue we did get 10x improvement -- if -- we hold the size & complexity of the system constant. Only if we do the one thing Brooks says would work: Buy vs. Build. When we write from scratch -- no way. Even then, I think that while we may see a 10x reduction for specific simple task, we won't see it for large, complex software, which is where most of the effort in software is invested.
      - jasode 6y ago
        >, Visual Basic and Access. Excel was released in 1987, and we had spreadsheets in the early '80s, too. When generalised, Brooks's prediction amounts to diminishing returns due to reduction of accidental complexity, and we're seeing exactly that. The "diminishing returns" of _what_ exactly? That's what I'm trying to make clear. Let me try and restate another way: (1) 10x improvement in programming tasks vs (2) 10x improvement in completing business projects I'm emphasizing that (1) has been achieved many times in multiple areas but it's overshadowed by not seeing (2) happen. I previously mentioned some things that I'm more than 10x faster on now: https://news.ycombinator.com/item?id=23758199 https://news.ycombinator.com/item?id=23758199 Visual Basic is another good example. When I first used VB Winforms in 1990s, it was more than 10x faster than hand-coding the raw C "Wndproc()" message loop. But that legitimate productivity gain is dwarfed by the business wanting new complexity (the app needs to connect to the internet, it needs to be new-fangled web app for browsers instead of a desktop exe, it needs to work on mobile phones, etc, etc). Our new desires for new business functionality multiply faster than the time-savings progress in tools. And "accidental complexity" isn't fixed either. And new deployment environments, new features also add a new set of accidental complexity. E.g. if next generation of apps need to interface to virtual reality (headsets, etc), the programming code will have logic that doesn't have direct business value. So we'll then get new 10x programming tool/library to manage that accidental complexity in the VR environment but then.... we're on to the neural implants SDK and we have no silver bullets for that new thing which means we revisit this topic again. >while we may see a 10x reduction for specific simple task, we won't see it for large, complex software, which is where most of the effort in software is invested. I agree. But again to be clear, today's expectation of "large, complex software" -- has also changed. EDIT reply to: "I'm saying that (1) has not been achieved even within a period of time that's 3x Brooks's prediction, " Raw Windows SDK C language WndProc() was late 1980s and by 1993, I was using Visual Basic 3 drag & drop buttons on to Winforms. Just that one example was 10x improvement within a decade. For line-of-business apps, VB was 10x+ more productive because of the paradigm shift (in addition to things like not worrying about mental bookkeeping of malloc()/free() etc.) >But most tasks cannot be achieved today 10x faster than in 1986 For discussion purposes, I don't know why we have to constantly refer to 1986 even though the paper has that date. It's repeated submission for discussion makes it seem like people consider it an evergreen topic that transcends Brook's days of the IBM 360 mainframe. As another example, the productivity improvement is the writing and deploying complex apps using Ruby on Rails or Javascript frameworks and deployed on AWS. That's more than is more 10x faster than the 1990s CGI days of having C Language code writing to stdout to output HTML. Those early web apps were simpler and yet they were so utterly tedious and slow to code.
- paulryanrogers 6y ago
  Total complexity must then exclude the layers of hardware, cloud infrastructure, and FOSS contributions that involved many other people.
pron 6y ago
I normally agree with Dan's essays, but this time I very much disagree. As someone who first started programming around the time Brooks's article was written, my conclusion is that not only was Brooks right, but, if anything, that he was too optimistic. In the late 80s and early 90s, a lot of software development was done with various RAD languages (Magic, Business Basic -- my first programming job used that -- and later VB). We fully expected the then-nascent Smalltalk, or something like it, to become dominant soon; does anyone claim that JS or Python are a significant improvement over it? Our current state is a big disappointment compared to where most programmers in the 80s believed we would be by now, and very much in line with Brooks's pouring of cold water (perhaps with the single exception of the internet). My perception is that the total productivity boost over the past three decades is less than one order-of-magnitude (Brooks was overly careful to predict no 10x boost due to one improvement in language design or programming methodology within one decade), and almost all of it comes from improvements in hardware and the online availability of free libraries (Brooks's "Buy vs Build", which he considered promising) and information -- not from changes in programming methodology or language design (although garbage collection and automated unit-tests have certainly helped, too). The article also mixes hardware improvements and their relationship to languages, but we knew that was going to happen back then, and I think it's been factored well into Brooks's prediction. Moreover, my perception is that we’re in a period of diminishing returns from languages, and that improvements to productivity Fortran and C had over Assembly are significantly greater than the gains since. The best way to judge Brooks's prediction, I think, is in comparison to opposite predictions made at the time -- like those that claimed Brooks's predictions were pessimistic -- and those were even more wrong in retrospect. I would also add that if you want to compare the ratio of essential and accidental complexity in line with Brooks's prescient analysis, you should compare the difficulty of designing a system in an accidental-complexity-free specification language like TLA+ to the difficulty of implementing it in a programming language from scratch. I find the claim that this ratio has improved by even one order of magnitude, let alone several, to be dubious. > Brooks states a bound on how much programmer productivity can improve. But, in practice, to state this bound correctly, one would have to be able to conceive of problems that no one would reasonably attempt to solve due to the amount of friction involved in solving the problem with current technologies. I don't think so. Although he stated it in practical terms, Brooks was careful to make a rather theoretical claim -- one that's supported by computational complexity results obtained in the 80s, 90s and 00s, on the hardness of program analysis -- about the ability to express what it is that a program is supposed to do.
tl 6y ago
What Dan got right: Brooks clearly missed the boat when he said, "Well how many MIPS can one use fruitfully?". I'm not sure where the fault lies on that one. A recent article [1] bemoans the slowness of modern software. What Dan got wrong: Both problems discussed were created by computers. A large collection of machines generating logs is possible to analyze, but that possibility comes from massive investment in tools designed to parse, extract and report on linear and structure data. Brooks covers this in "Buy vs. Build", although his example is closer to Excel than ggplot. Also, if you are dashing off a python script in a single wall-clock minute, you either write insane amounts of Python for purposes similar to this or are way smarter than the average engineer. In Advent of Code (where Python is popular), the fastest solution to the easiest puzzle (Day 1) this year was 0:07:11. [2] [1] https://news.ycombinator.com/item?id=25386098 https://news.ycombinator.com/item?id=25386098 [2] https://adventofcode.com/2020/leaderboard/day/1 https://adventofcode.com/2020/leaderboard/day/1
- Olreich 6y ago
  I just tracked my time for doing that Advent of Code day 1. It took 10 minutes from start to finish. Across the two parts of the question, I spent approximately 2 minutes building the scripts, 1 minute determining an algorithm, and 7 minutes reading the prompt. I suspect similar breakdowns for the 7:11 you quote. If Dan excludes the prompt reading and solution determination because he's seen this sort of problem before, 1 minute using tools he's comfortable with doesn't seem unreasonable. Considering all it is is a list of hosts, a loop, a try/except block, and a scp command, using much more than a couple of minutes writing would be surprising to me.
  - Jtsummers 6y ago
    There was a 6 minute delay in getting the inputs for day one for many people. The fastest part one solution was 35 seconds. And the fastest part two (after finishing part one) was also about 30 seconds. If there hadn’t been a server issue I suspect many of the top 100 would’ve finished both parts in under two minutes
    - 6y ago
amadeuspagel 6y ago
Technological improvements can also make it easier to deal with conceptual complexity, by allowing us to express ourselves better.
coldcode 6y ago
Complexity in just what I and doing right now is impossible to comprehend. Imagine a mobile app with dozens of dependencies taking on a "secret" project involving a dozen of those dependencies that are drifting far way from the rest, with a dependency manager unable to function in this circumstance requiring manual guessing of what goes with what. Now imagine trying to develop with this Frankenstein set of dependencies and have any hope of getting work done. Of course the problem is not technical, its institutional; yet it's an unreasonable well of complexity that belies the demands placed on its workers. Technological complexity is not the only kind to worry about, it's the difficulties of managing complexities that are not obvious and cannot be explained to non technology savvy executives leading to painful results. Now add a wealth of server side complexities with webs of micro and not-so-micro services interdependent on each other made by teams further and further removed from each other leading to more instability and unpredictable behaviors. Managing systemic complexity (or not doing it well) seems endemic to programming today much worse than when Brooks wrote his essay (and I actually predate).
CraigJPerry 6y ago
> In 1986, perhaps I would have used telnet or ftp instead of scp. In 1986 you would probably have used rcp (the thing that inspired scp). A lot less secure but with one benefit - when Dan did this mass scp i bet his local CPU was absolutely pegged well before network saturation? Ssh imposes a hefty cpu overhead because of encryption that i didn’t appreciate until around 2005 when i was updating an rcp centric script to use scp. > Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today You could have used ksh (a popular scriptable shell that heavily inspired Perl). rcp errors are denoted by return code and thats also the natural error handling approach in ksh. As for the grep / sed / awk part of the problem, it’s possible the 1986 local machine was uniprocessor - it might have been faster to do the processing locally on each node (via rsh) then only network transfer the result set back to the local node. EDIT - job control (the thing that makes parallelism trivial in ksh) was only added in 1989. To easily get parallelism you would have had to write the script in csh - life’s too short for scripting in csh (or its descendants) so i conclude everything i said before is wrong and in 1986 i’d have said sod this and went and brewed a cuppa tea instead :-) (not strictly true since i was coming up on 2 years old at that time...)
- earthboundkid 6y ago
  Yeah, my thought on the problem was that there’s probably an AT&T promo video from 1986 on YouTube somewhere that shows Brian Kernighan logging into 10 machines and downloading phone records and then processing them with AWK. He just wouldn’t be able to download quite as many and do as much with it.
mikewarot 6y ago
In the advent of code, I solved day 23 (the crab and the cups) doing brute force, and even with today's hardware it would have taken many hours, if not days, to complete. A tweak of the algorithm got it down to 0.248 seconds. That tweak required an 4 megabyte array in RAM. I back ported the solution to Turbo Pascal 7, it used a "file of longint" in place of the array which you can't do in MS-DOs... it finished in about 10 minutes, because I have an SSD. Otherwise it would have been about 50,000,000 IO operations at 10/second --> 50+ days. We can and do use the heck out of the hardware, but it's not infinite, and there will always be orders of magnitude performance to be gained by seeking and using better algorithms.
- mark254 6y ago
  Well, using "himem.sys" you could make use of RAM above 1MB... I used it to write a sort utility that only uses temp files when RAM was exhausted. On my 386 with 4MB total RAM I could use about 3.7 or so, the rest was a shadowed BIOS copy. (Never tried to deactivate that, perhaps it would have been relocated and then made available?)
hbrundage 6y ago
A missing colour in the discussions around complexity today is the ever growing ambition for the problems we're all trying to solve. My grand unified profoundly unserious theory is that we're always going to complain about accidental complexity because we will keep discovering new reams of it by extending what we're trying to do out into the unknown parts. If we continued to play in just the known parts of the problem space, we'd see it melt away further and further into Brooks' model of the world. Instead, we built distributed systems, we work in absolutely giant teams, we deploy to the strange heterogeneous runtime we call the web, etc etc, and all of that generates new complexities to abstract away before we get back to just the essential stuff.
eternalban 6y ago
> In this example, let's say that we somehow had enough storage to keep the data we want to query in 1986. The next part would be to marshall on the order of 1 CPU-year worth of resources and have the query complete in minutes. As with > the storage problem, this would have also been absurd in [1986], so we've run into a second piece of non-essential complexity so large that it would stop a person from 1986 from thinking of this problem at all.* If you are given a bucket to drain a lake, the enormity of task at hand is not "accidental complexity". OP is asserting that doing x one time or a billion times is still x, with an inherent complexity of using a bucket to (dip, draw, transfer, drain). This is true -only- if we willfully insist on ignoring that our 'conceptual model' (a reservoire, a bucket, and a worker) is intimately bound with available technology and is a leaky abstraction. The actual conceptual [model] (move liquid from one reservoire to another) says very little about "inherent complexity". (Can you point it out?) It seems it is more accurate to note that the complexity of materializing abstractions tends to increase with scale of operations. Our goal then, as a discipline, should be to insure this relationship approaches linear [or better] to the extent possible. Continuing the example, the actual task (drain a lake) could materialize as an army of workers, each with a bucket, all converging on a sink to drain their bucket. Is managing the workers really an "accidental" complexity? What above then implies is that the notion of "essential complexity" is a faulty paradigm if considered to be a static measure indepdendent of du jour technology. The essential complexity of a task is held to be inherently related to the magnitude of task, and thus inherently addressable by, and related to, advancements in technology. And finally, since nearly all of the modern software systems are merely a scaled up ("web scale") version of earlier, "less complex", systems, it seems reasonable to question the industry's singular obsession with languages as the remedy to what is inherently a problem of scaling well understood tasks.
eadmund 6y ago
> In 1986, perhaps I would have used telnet or ftp instead of scp. Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today. In fact, I think just getting semi-decent error handling while managing a connection pool could have easily taken an order of magnitude longer than this entire task took me (not including time spent downloading logs in the background). > Today, we have our choice of high-performance languages where it's easy to write, fast, safe code > In 1986, there would have been no comparable language Shell existed in 1986, and IIRC ftp could actually be scripted, a bit, so I think that it could have been used to run the download portion. Maybe I misremember? Lisp existed in 1986; the first edition of Common Lisp the Language was written in 1984. It certainly could have been used, and even in the 80s I think it would have good enough performance for log parsing. Certainly it would now, but that is after four decades of optimisation. His point about libraries, though, is excellent. The rise of open source libraries radically changed the software world for the better. I should note, though, that Plan 9 — which was developed in the late 80s — removed a ton of the accidental complexity in dealing with e.g. FTP servers. We could have much less accidental complexity, but we choose not to.
- pydry 6y ago
  >His point about libraries, though, is excellent. The rise of open source libraries radically changed the software world for the better. This was one of the first points made in no silver bullet (except he said buy not "download for free"). It was a lot less obvious in 1986 than it is today and arguably it's not even all that obvious today.
- kazinator 6y ago
  The reality is that Common Lisp would not fit onto a Unix workstation from 1986 in a light weight way, like firing off another /bin/sh or /bin/awk job. Machines had memories measured in single digit megabytes, often closer to zero than to 10.
karmakaze 6y ago
The fault in this discussion and what I was hoping to learn about is the reduction in complexity of evolving systems. This post and the original cited work is very much about one-shot waterfall development. This is not what is done today and has little bearing. How much of the effort in fitting a new requirement into an existing system is due to choices in the way it was decomposed and structured? I'm convinced there are much better than 2x better ways of keeping this effort low by choosing seams carefully and putting in the work when some are found to be in messy places. Another way to say this might be, any system could (for sake of argument) be built with 2x essential complexity, but an evolved system typically has a much higher factor due to its lineage. Finding ways of keeping this near 2x should be the focus.
taeric 6y ago
My gut is that the claim translates better to percentage of processing power per user. With such an excess of power today, you can scrape a ton of use out of it. That said, the use of scripting languages kind of proves the point. There is little attempt made to maximize compute. Only gets more true when you consider the absurd amount of memory being used.
pydry 6y ago
Fred Brooks suggests in 1986 that future productivity improvements focus more on: >Exploiting the mass market to avoid constructing what can be bought In 2020 danluu says that: >In 1986, there would have been no comparable language, but more importantly, I wouldn't have been able to trivially find, download, and compile the appropriate libraries and would've had to write all of the parsing code by hand, turning a task that took a few minutes into a task that I'd be lucky to get done in an hour. IME this demonstrates remarkable prescience by Brooks. The only real difference between Brooks' hypothetical future where software is assembled from bought components and the one danluu lives in is that the components are, in this case, free. Package managers might seem obvious and necessary today, but I don't think they were at all obvious in 1986 (or even in 1996). Fuck, golang didn't even think they were necessary in 2009. Also: >Looking at the specifics with the benefit of hindsight, we can see that Brooks' 1986 claim that we've basically captured all the productivity gains I don't believe that this was his claim. He claimed that productivity gains from improved programming language design would be incremental rather than game changing (hence: no silver bullet). Some people probably will argue that there is a 10x difference in language productivity due to language but I don't think that is the case. In fact, I think there's a class of people who keep getting proved wrong and wrong again about the relative importance of package management ecosystems over "game changing" language features like non nullable types that are incrementally better, but not an order of magnitude better.
- ncmncm 6y ago
  I see that you are unaware of the large collection of programming language improvements accumulated in recent decades, which I will guess is because your favored language mostly does not implement them.
  - pydry 6y ago
    Which improvements caused a 2x+ productivity boost that my favored language does not have?
    - ncmncm 6y ago
      Guessing again based on your nick, improvements that draw on type theory or type mechanics.
BenoitEssiambre 6y ago
This is a pretty poor argument. This does not represent productivity in solving new problems, it just tells us it's worth reusing pieces for already solved problems. Yeah carpenters might seem very productive if they started with increasingly already built houses like programmers do. Solving the part of the problem that is novel still has irreducible complexity.
- coldtea 6y ago
  >This does not represent productivity in solving new problems, it just tells us it's worth reusing pieces for already solved problems. That's neither here nor there. First because most (if not all) of new problems can be solved by reusing existing parts (from languages and libs to widgets and cli tools). Second, because we tend to re-work on the same problems over and over for the most part, anyway, so improvements on those are still greatly important (even more so than novel problems actually, which are about potential uncertain future markets, whereas solved problems are what drive existing global markets, so dozens of trillions of dollars).
vannevar 6y ago
I don't think that Brooks was arguing that any given fixed programming task could not be made easier over time; clearly, even in 1986, there were programming tasks that had become trivial by use of existing code but were originally quite challenging to code on their own. I think Brooks was talking about the work of the individual programmer: given two identical programmers, with the same available libraries of existing code, is there a language or IDE that will make one 10x more productive than the other? And I think the answer is still, no.
- brabel 6y ago
  Exactly, this blog post missed the point completely. Unbelievable.
- joshuafcole 6y ago
  I think the lines between "available library of existing code", language, and IDE are much murkier than they might appear. If one programmer is given raw assembly and the other Python, you're probably going to see at _least_ a 10x productivity bump to the python user _even if they use no additional libraries_. Language and IDE choices impact the vocabulary available to express solutions in the same way.
  - vannevar 6y ago
    You can certainly contrive a counter-example---even in 1986, you could say much the same thing about assembly and something like C or BASIC. But once that leap of abstraction was made, we seem to have hit a plateau of expressive power. Is the jump from C to Rust as empowering as the jump from assembly to C? I don't think it is.
- jnwatson 6y ago
  You're added premise is an implicit point of Luu's article. Or perhaps the point is that Brooks' framing doesn't really apply when you can avoid programming altogether. The silver bullet is open source. The ability to utilize the millions of hours of other people's work to accomplish your programming task is the game changer, and is what makes formerly complicated tasks trivial. The essential complexity of the software I deal with is about two orders of magnitude more complicated than stuff I dealt with 20 years ago. I'm not sure that 20 years from now it will be another two orders of magnitude, but perhaps with utility computing and higher level orchestration concepts, it very well might be.
ozim 6y ago
I have different idea about essential complexity and accidental complexity. I think examples in the article are all just accidental complexity. Essential complexity from how I read Fred Brooks is a domain knowledge like what an accountant does and what features he is using. All frameworks, servers and doing scp or queries in that is accidental complexity. Accountant does not need any React or Angular front end, they don't need a server. Where angular brings in reduction in accidental complexity since by using it I don't have to build a lot of stuff from scratch, it does not make tax laws any simpler. Where tax laws are essential complexity that needs to be coded in accounting software. The same with servers, going to cloud so reducing my accidental complexity by removing need to deal with servers is not making tax laws simpler or easier to grasp. This way in my opinion, essential complexity and accidental complexity holds really well as general idea.
- sanxiyn 6y ago
  Eh, you and Dan are in agreement? "This task ... is still nearly entirely accidental complexity", etc.
  - ozim 6y ago
    Not really, because I think that tasks he picked as "essential complexity" are still fully "accidental complexity" in terms of how I understand Brooks writing. I cannot agree with Dan, because I feel that all "computer stuff" is accidental complexity in Brooks writing context. Some other replies change context into "from software dev perspective that is an essential complexity", now we are discussing what Fred Brooks wrote in 1987. If we change that context then I can agree with Dan, but if you want to refute argument you have to operate inside the same context. What I see that Dan argues, is accidental complexity is not getting easier throughout the years. He does not touch anything about essential complexity being well "complex" and that we cannot do much about it. So I don't agree that you can refute whole idea based on those examples.
    - sanxiyn 6y ago
      Where does Dan pick anything as essential complexity? Please cite quote. As far as I can tell, Dan considers everything in the article as accidential complexity, just like you do (and I do).
dustingetz 6y ago
Hypothesis: Constraints on time is the cause of accidental complexity. Given infinite time I would think longer, yak shave more, rewrite everything simpler, recur until perfect. In real life what constrains time? The business, its competition and available resources. Traded off vs speed of human thought and communication.
deleted 6y ago
[deleted]
heisenbit 6y ago
Brooks: > The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems. I think Brooks by focusing on the reality of the world reflected into topology of systems focuses on the essentials for the complexity. Yes, we can abstract some complexity but we also know these abstractions are leaky - heaping too many of them on top of each other and it topples. Even more important there are limits to how much most of us can juggle in their minds - exceed that threshold and we need a team with all the overhead that accompanies that. Once inter human communication is part of the process any optimization on tool level can only improve the situation by so much.
neolog 6y ago
Dan, I'm curious how all those people listed at the bottom read the post early. I sometimes want early feedback on my writing and you seem to be getting it from a lot of well known people. How do you do that? Do you just email them all and ask?
astine 6y ago
This essay badly misinterprets Brooks original essay. Brooks was talking specifically about the problem of building new systems from scratch.He was speaking about programming productivity. This essay is talking about operational tasks. Of course operational tasks can be speed up with improvements to software and hardware, that's the whole point of writing new software and developing new hardware. Furthermore, Brooks doesn't actually argue that there is no way to tackle essential complexity when writing new software. What he says is that there is no single technique that will speed up the process of writing code across the board, that is, there is no silver bullet. However, he does make allowance for techniques that allow us to break up complex systems into smaller systems which can then be solved independently. He specifically mentions object oriented programming as a tool which makes it easier for programmers to write software as components but CPAN would have been a better example. He just doesn't allow that OOP by itself would allow for an order of magnitute improvement (something which I think that experience has born out.) The author of this essay, points out a couple of operational tasks which involve programming were easier for them than they would have been 34 years ago because faster hardware noow exists and because software that already takes care of a subset of the problem now exists when it didn't before. Partially, this is confirmation of Brooks' idea that breaking down problems it the way to go long-term, but more critically, it's a failure to realize on the part of the author that they're speaking of productivity in their role as the user of a computer system whereas Brooks is speaking about productivity in designing new software products. edit: I think that the fundamental root of the essay's problem is that the author seems to have misssed what Brooks was referring to which he discussed accidental vs essential complexity. Brooks specifically pointed out that he was referencing Aristotle when he used those terms, and in this case essential doesn't mean unavoidable, but a part of the essential character of the problem itself. A more modern terminology would be intrinsic vs extrinsic. Complexity is extrinsic to programming if it comes from something other than the nature of the problem itself. If my computer is too slow to allow me to write a program, then that's probably not due the the nature of the program itself (except indirectly) and so represents a form of accidental complexity. Feature requirements however, represent essential complexity. You can manage essential complexity according to Brooks by dividing the problem up or avoiding altogether, but it's not something that will go away simply with better tooling or a better methodology, (unless that better tooling replaces software that you otherwise would have written.
zmmmmm 6y ago
I found the example given very contrived and the whole article slightly disingenuous because of it. Brooks was talking about building long-living applications serving complex business requirements. The article talks about a one off script with no ongoing life. It's not even really "software" at all in the terms Brooks was describing. It's like someone describing absolute limits on fuel efficiency for vehicles and then I get on a scooter and coast down a hill and jump off at the end so that it crashes into a wall and explodes, and using this example to claim they are wrong.
- daotoad 6y ago
  The difference between a one off script and a production-grade system with documentation, instrumentation, and monitoring is pretty big. Here's less than a minute's worth of questions I would want answered for a professional version of the log fetching example: * Is that list of a thousand servers static? * How is it updated? * What format is it in? * What do you do when the resource isn't available? * What other systems are impacted when the resource can't be contacted? * Is it acceptable to use a cached server list until it the resource can be updated? * How do you handle servers that don't respond when polled? If we retry connections, what scheme do we use? Linear or exponential back off? * After a service interruption, how do we resume data collection? Do we keep rolling forward or do we request older data? How quickly do we retrieve backed up data? Is service resumption plan different for the case when a single system is down vs when the log fetching system has an outage?
- mchusma 6y ago
  Agree. Nearly all the work of programming I have done is related to the task of trying to figure out the business problem in enough extreme detail to provide a solution. The main lever to improve productivity for me as a developer are open source tools, commercial APIs, and similar things. This isn't reducing the amount of programming, it's just moving it around. Magically 1,000x faster computers would maybe help me be 20% more productive.
0xbadcafebee 6y ago
Let's say you're building a home. Foundation is poured and it's time to frame it. If you hire some random guys with power tools standing outside a home depot, it'll take weeks to months for them to finish, and might well end up crappy. But hire two 55-year-old men with hammers, nails and a single skil-saw, and they can have it up in 3-4 days. Or grab your Amish neighbors and raise a barn in a day. Productivity is not bounded by your access to technology, it's bounded by the problem you need to solve and the means you use to solve it. > Brooks explicitly dismisses increased computational power as something that will not improve productivity ("Well, how many MIPS can one use fruitfully?", more on this later), but both storage and CPU power (not to mention network speed and RAM) were sources of accidental complexity so large that they bounded the space of problems Brooks was able to conceive of. Yet here in the future, our excess of storage, CPU power and network bandwidth have bounded our ability to be fruitful with them. If you add up the MIPS we use today and compare it to what solutions we're implementing with it, we are vastly less productive today. Brooks was right, but in a weird way. There is a bound on programmer productivity, and it is the programmer. In the past, people wouldn't have set themselves up to need to collect a billion billion logs to analyze some tiny metric among them. Anyone would've seen that as poor planning and a horrible waste of resources. Why would you bury needles in billions of haystacks, and then need to hastily build a robot that finds needles? Why not just keep the needles in a pile of needles? Do you really even need all those needles? > Thinking that the tools you personally use are as good as it gets is an easy trap to fall into. This actually seems like a hard trap for most to fall into. Nearly every software engineer I've met either thinks they could make a better tool, or just wants to try for the hell of it. On the other hand, most engineers think that building a better mouse trap will solve all their mouse problems.
ncmncm 6y ago
The only way Brooks's argument makes sense is that we naturally tackle problems as big as the resources available to us permit. So, our daily load of programming hardly changes, because as we get better equipped, more is asked for. But in that interpretation, none of his supporting detail applies.
sontek 6y ago
Maybe off topic but this site might have convinced me to move to Firefox. Are all of you reading this page on mobile or a small screen? I have a large monitor and this page expands full width of the screen and the lines are extremely hard to follow. It turns out Firefox has a "Reader View" that makes sites like this be put into a nice readable column. Here is a comparison of the two: https://i.imgur.com/plV0e7l.png https://i.imgur.com/plV0e7l.png I find it interesting that this post reached to the top of HN with such an unreadable layout and so I was interested in if everyone has some tricks for reading content like this or is Firefox/Reader Mode the best way?
konjin 6y ago
Going to hard disagree. What he's saying there is that today we have free implementations of libraries which solve incidental complexity and those libraries work (nearly) everywhere (nearly) the same. I worked in finance on K and the name makes in impossible to use google/stack overflow to find anything about the language. Which meant I had a paid for help and tech support from a company that's been largely unchanged from the 80s. It was fucking phenomenal. Once I gave up the idea that I knew best and started working in the way the language expected me my productivity was an order of magnitude higher than it would have been in a cobbled together python pipeline like the one he uses there. And because someone was responsible for the language and tooling I could use scripts from before I was in primary school, and they worked as expected. I dare you to try and do the same with any language that isn't C.

246 points by weinzierl 6y ago

It is easy to see that the total complexity has been severely reduced in the last years when you look at how many small companies are doing things that required an army of people a few decades ago. For example, running the technology for a real-time banking system nowadays can be done by perhaps even a single developer (with enough experience) thanks to open source, cloud computing and advances in software and hardware. Now we even have distributed relational databases that are relatively straightforward to operate, like CockroachDB.

pron 6y ago
First, Brooks wasn't talking about that. He was talking about productivity boosts due to programming language design and programming methodology. More specifically, he did not exclude improvements to both kinds of complexity overall, but only those that drastically change their relative proportion. Second, in the early 90s I had a summer job at a company whose old ERP, which had been written in a language called Business Basic and ran on a minicomputer, was maintained and operated by a single person.
- jasode 6y ago
  >, Brooks wasn't talking about that. He was talking about productivity boosts due to programming language design and programming methodology. Brooks was also talking about non-programming language advancements as a possible "silver bullets". See the pdf and look for the following sections that are not about programming syntax: Hopes for the Silver Artificial Intelligence Environments and Tools Promising Attacks on the Conceptual Essence Buy versus Build He underestimated AI machine learning. With hindsight, we see that the deep-layer neural net combined with GPU beat human rule-based programming for automatic language translations, DeepMind AlphaZero beats expert programmers hand-tweaking IBM DeepBlue, etc. EDIT reply to : ", nor have they reduced the overall cost of programming by 10x even after more than three decades." I'm not convinced of that. The issue with your conclusion is that it omits the increasing expectations of more complex systems. I'd argue we did get 10x improvement -- if -- we hold the size & complexity of the system constant. E.g. write 1970s style text-based code to summarize sales revenue by region and output it to green bar dot matrix printers. This was tedious work in old COBOL but much easier today with Python Pandas or even no code tools like MS Excel. The invisible factor we often overlook is that our demand for software to do more complex things will always outpace the smaller productivity increases of the tools. This makes it look like our newer programming tools never gave us 10x improvement when it actually did.
  - pron 6y ago
    How did he underestimate statistical machine learning? Whatever achievements were made, they did not take place within a decade, nor have they reduced the overall cost of programming by 10x even after more than three decades. And, indeed, the one thing that Brooks presented as being the most promising direction, i.e. buy vs. build. He said that unlike changes to programming languages and methodology that wouldn't give a huge boost, buy vs. build might (although he wasn't sure about that). So he was exactly right about that. Also, 1986 wasn't 1966. In 1986 people didn't write simple software in COBOL. They used things like Magic and Business Basic and even Smalltalk, and, shortly after, Visual Basic and Access. Excel was released in 1987, and we had spreadsheets in the early 80s, too (and Brooks explicitly mentions them in No Silver Bullet). RAD tools and "no code" was very much in vogue in the late 80s and early 90s. That was Brooks's present, not future. He even says that this is the right direction: I believe the single most powerful software-productivity strategy for many organizations today is to equip the computer-naive intellectual workers who are on the firing line with personal computers and good generalized writing, drawing, file, and spreadsheet programs and then to turn them loose. The same strategy, carried out with generalized mathematical and statistical packages and some simple programming capabilities, will also work for hundreds of laboratory scientists. When generalised, Brooks's prediction amounts to expecting diminishing returns due to reduction of accidental complexity, and we're seeing exactly that. > I'd argue we did get 10x improvement -- if -- we hold the size & complexity of the system constant. Only if we do the one thing Brooks says would work: Buy vs. Build. When we write from scratch -- no way. Even then, I think that while we may see a 10x reduction for specific simple task, we won't see it for large, complex software, which is where most of the effort in software is invested.
    - jasode 6y ago
      >, Visual Basic and Access. Excel was released in 1987, and we had spreadsheets in the early '80s, too. When generalised, Brooks's prediction amounts to diminishing returns due to reduction of accidental complexity, and we're seeing exactly that. The "diminishing returns" of _what_ exactly? That's what I'm trying to make clear. Let me try and restate another way: (1) 10x improvement in programming tasks vs (2) 10x improvement in completing business projects I'm emphasizing that (1) has been achieved many times in multiple areas but it's overshadowed by not seeing (2) happen. I previously mentioned some things that I'm more than 10x faster on now: https://news.ycombinator.com/item?id=23758199 https://news.ycombinator.com/item?id=23758199 Visual Basic is another good example. When I first used VB Winforms in 1990s, it was more than 10x faster than hand-coding the raw C "Wndproc()" message loop. But that legitimate productivity gain is dwarfed by the business wanting new complexity (the app needs to connect to the internet, it needs to be new-fangled web app for browsers instead of a desktop exe, it needs to work on mobile phones, etc, etc). Our new desires for new business functionality multiply faster than the time-savings progress in tools. And "accidental complexity" isn't fixed either. And new deployment environments, new features also add a new set of accidental complexity. E.g. if next generation of apps need to interface to virtual reality (headsets, etc), the programming code will have logic that doesn't have direct business value. So we'll then get new 10x programming tool/library to manage that accidental complexity in the VR environment but then.... we're on to the neural implants SDK and we have no silver bullets for that new thing which means we revisit this topic again. >while we may see a 10x reduction for specific simple task, we won't see it for large, complex software, which is where most of the effort in software is invested. I agree. But again to be clear, today's expectation of "large, complex software" -- has also changed. EDIT reply to: "I'm saying that (1) has not been achieved even within a period of time that's 3x Brooks's prediction, " Raw Windows SDK C language WndProc() was late 1980s and by 1993, I was using Visual Basic 3 drag & drop buttons on to Winforms. Just that one example was 10x improvement within a decade. For line-of-business apps, VB was 10x+ more productive because of the paradigm shift (in addition to things like not worrying about mental bookkeeping of malloc()/free() etc.) >But most tasks cannot be achieved today 10x faster than in 1986 For discussion purposes, I don't know why we have to constantly refer to 1986 even though the paper has that date. It's repeated submission for discussion makes it seem like people consider it an evergreen topic that transcends Brook's days of the IBM 360 mainframe. As another example, the productivity improvement is the writing and deploying complex apps using Ruby on Rails or Javascript frameworks and deployed on AWS. That's more than is more 10x faster than the 1990s CGI days of having C Language code writing to stdout to output HTML. Those early web apps were simpler and yet they were so utterly tedious and slow to code.
paulryanrogers 6y ago
Total complexity must then exclude the layers of hardware, cloud infrastructure, and FOSS contributions that involved many other people.

pron 6y ago

I normally agree with Dan's essays, but this time I very much disagree. As someone who first started programming around the time Brooks's article was written, my conclusion is that not only was Brooks right, but, if anything, that he was too optimistic. In the late 80s and early 90s, a lot of software development was done with various RAD languages (Magic, Business Basic -- my first programming job used that -- and later VB). We fully expected the then-nascent Smalltalk, or something like it, to become dominant soon; does anyone claim that JS or Python are a significant improvement over it? Our current state is a big disappointment compared to where most programmers in the 80s believed we would be by now, and very much in line with Brooks's pouring of cold water (perhaps with the single exception of the internet). My perception is that the total productivity boost over the past three decades is less than one order-of-magnitude (Brooks was overly careful to predict no 10x boost due to one improvement in language design or programming methodology within one decade), and almost all of it comes from improvements in hardware and the online availability of free libraries (Brooks's "Buy vs Build", which he considered promising) and information -- not from changes in programming methodology or language design (although garbage collection and automated unit-tests have certainly helped, too). The article also mixes hardware improvements and their relationship to languages, but we knew that was going to happen back then, and I think it's been factored well into Brooks's prediction. Moreover, my perception is that we’re in a period of diminishing returns from languages, and that improvements to productivity Fortran and C had over Assembly are significantly greater than the gains since. The best way to judge Brooks's prediction, I think, is in comparison to opposite predictions made at the time -- like those that claimed Brooks's predictions were pessimistic -- and those were even more wrong in retrospect. I would also add that if you want to compare the ratio of essential and accidental complexity in line with Brooks's prescient analysis, you should compare the difficulty of designing a system in an accidental-complexity-free specification language like TLA+ to the difficulty of implementing it in a programming language from scratch. I find the claim that this ratio has improved by even one order of magnitude, let alone several, to be dubious. > Brooks states a bound on how much programmer productivity can improve. But, in practice, to state this bound correctly, one would have to be able to conceive of problems that no one would reasonably attempt to solve due to the amount of friction involved in solving the problem with current technologies. I don't think so. Although he stated it in practical terms, Brooks was careful to make a rather theoretical claim -- one that's supported by computational complexity results obtained in the 80s, 90s and 00s, on the hardness of program analysis -- about the ability to express what it is that a program is supposed to do.

tl 6y ago

What Dan got right: Brooks clearly missed the boat when he said, "Well how many MIPS can one use fruitfully?". I'm not sure where the fault lies on that one. A recent article [1] bemoans the slowness of modern software. What Dan got wrong: Both problems discussed were created by computers. A large collection of machines generating logs is possible to analyze, but that possibility comes from massive investment in tools designed to parse, extract and report on linear and structure data. Brooks covers this in "Buy vs. Build", although his example is closer to Excel than ggplot. Also, if you are dashing off a python script in a single wall-clock minute, you either write insane amounts of Python for purposes similar to this or are way smarter than the average engineer. In Advent of Code (where Python is popular), the fastest solution to the easiest puzzle (Day 1) this year was 0:07:11. [2] [1] https://news.ycombinator.com/item?id=25386098 https://news.ycombinator.com/item?id=25386098 [2] https://adventofcode.com/2020/leaderboard/day/1 https://adventofcode.com/2020/leaderboard/day/1

Olreich 6y ago
I just tracked my time for doing that Advent of Code day 1. It took 10 minutes from start to finish. Across the two parts of the question, I spent approximately 2 minutes building the scripts, 1 minute determining an algorithm, and 7 minutes reading the prompt. I suspect similar breakdowns for the 7:11 you quote. If Dan excludes the prompt reading and solution determination because he's seen this sort of problem before, 1 minute using tools he's comfortable with doesn't seem unreasonable. Considering all it is is a list of hosts, a loop, a try/except block, and a scp command, using much more than a couple of minutes writing would be surprising to me.
- Jtsummers 6y ago
  There was a 6 minute delay in getting the inputs for day one for many people. The fastest part one solution was 35 seconds. And the fastest part two (after finishing part one) was also about 30 seconds. If there hadn’t been a server issue I suspect many of the top 100 would’ve finished both parts in under two minutes
  - 6y ago

amadeuspagel 6y ago

Technological improvements can also make it easier to deal with conceptual complexity, by allowing us to express ourselves better.

coldcode 6y ago

Complexity in just what I and doing right now is impossible to comprehend. Imagine a mobile app with dozens of dependencies taking on a "secret" project involving a dozen of those dependencies that are drifting far way from the rest, with a dependency manager unable to function in this circumstance requiring manual guessing of what goes with what. Now imagine trying to develop with this Frankenstein set of dependencies and have any hope of getting work done. Of course the problem is not technical, its institutional; yet it's an unreasonable well of complexity that belies the demands placed on its workers. Technological complexity is not the only kind to worry about, it's the difficulties of managing complexities that are not obvious and cannot be explained to non technology savvy executives leading to painful results. Now add a wealth of server side complexities with webs of micro and not-so-micro services interdependent on each other made by teams further and further removed from each other leading to more instability and unpredictable behaviors. Managing systemic complexity (or not doing it well) seems endemic to programming today much worse than when Brooks wrote his essay (and I actually predate).

CraigJPerry 6y ago

> In 1986, perhaps I would have used telnet or ftp instead of scp. In 1986 you would probably have used rcp (the thing that inspired scp). A lot less secure but with one benefit - when Dan did this mass scp i bet his local CPU was absolutely pegged well before network saturation? Ssh imposes a hefty cpu overhead because of encryption that i didn’t appreciate until around 2005 when i was updating an rcp centric script to use scp. > Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today You could have used ksh (a popular scriptable shell that heavily inspired Perl). rcp errors are denoted by return code and thats also the natural error handling approach in ksh. As for the grep / sed / awk part of the problem, it’s possible the 1986 local machine was uniprocessor - it might have been faster to do the processing locally on each node (via rsh) then only network transfer the result set back to the local node. EDIT - job control (the thing that makes parallelism trivial in ksh) was only added in 1989. To easily get parallelism you would have had to write the script in csh - life’s too short for scripting in csh (or its descendants) so i conclude everything i said before is wrong and in 1986 i’d have said sod this and went and brewed a cuppa tea instead :-) (not strictly true since i was coming up on 2 years old at that time...)

earthboundkid 6y ago
Yeah, my thought on the problem was that there’s probably an AT&T promo video from 1986 on YouTube somewhere that shows Brian Kernighan logging into 10 machines and downloading phone records and then processing them with AWK. He just wouldn’t be able to download quite as many and do as much with it.

mikewarot 6y ago

In the advent of code, I solved day 23 (the crab and the cups) doing brute force, and even with today's hardware it would have taken many hours, if not days, to complete. A tweak of the algorithm got it down to 0.248 seconds. That tweak required an 4 megabyte array in RAM. I back ported the solution to Turbo Pascal 7, it used a "file of longint" in place of the array which you can't do in MS-DOs... it finished in about 10 minutes, because I have an SSD. Otherwise it would have been about 50,000,000 IO operations at 10/second --> 50+ days. We can and do use the heck out of the hardware, but it's not infinite, and there will always be orders of magnitude performance to be gained by seeking and using better algorithms.

mark254 6y ago
Well, using "himem.sys" you could make use of RAM above 1MB... I used it to write a sort utility that only uses temp files when RAM was exhausted. On my 386 with 4MB total RAM I could use about 3.7 or so, the rest was a shadowed BIOS copy. (Never tried to deactivate that, perhaps it would have been relocated and then made available?)

hbrundage 6y ago

A missing colour in the discussions around complexity today is the ever growing ambition for the problems we're all trying to solve. My grand unified profoundly unserious theory is that we're always going to complain about accidental complexity because we will keep discovering new reams of it by extending what we're trying to do out into the unknown parts. If we continued to play in just the known parts of the problem space, we'd see it melt away further and further into Brooks' model of the world. Instead, we built distributed systems, we work in absolutely giant teams, we deploy to the strange heterogeneous runtime we call the web, etc etc, and all of that generates new complexities to abstract away before we get back to just the essential stuff.

eternalban 6y ago

> In this example, let's say that we somehow had enough storage to keep the data we want to query in 1986. The next part would be to marshall on the order of 1 CPU-year worth of resources and have the query complete in minutes. As with > the storage problem, this would have also been absurd in [1986], so we've run into a second piece of non-essential complexity so large that it would stop a person from 1986 from thinking of this problem at all.* If you are given a bucket to drain a lake, the enormity of task at hand is not "accidental complexity". OP is asserting that doing x one time or a billion times is still x, with an inherent complexity of using a bucket to (dip, draw, transfer, drain). This is true -only- if we willfully insist on ignoring that our 'conceptual model' (a reservoire, a bucket, and a worker) is intimately bound with available technology and is a leaky abstraction. The actual conceptual [model] (move liquid from one reservoire to another) says very little about "inherent complexity". (Can you point it out?) It seems it is more accurate to note that the complexity of materializing abstractions tends to increase with scale of operations. Our goal then, as a discipline, should be to insure this relationship approaches linear [or better] to the extent possible. Continuing the example, the actual task (drain a lake) could materialize as an army of workers, each with a bucket, all converging on a sink to drain their bucket. Is managing the workers really an "accidental" complexity? What above then implies is that the notion of "essential complexity" is a faulty paradigm if considered to be a static measure indepdendent of du jour technology. The essential complexity of a task is held to be inherently related to the magnitude of task, and thus inherently addressable by, and related to, advancements in technology. And finally, since nearly all of the modern software systems are merely a scaled up ("web scale") version of earlier, "less complex", systems, it seems reasonable to question the industry's singular obsession with languages as the remedy to what is inherently a problem of scaling well understood tasks.

eadmund 6y ago

> In 1986, perhaps I would have used telnet or ftp instead of scp. Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today. In fact, I think just getting semi-decent error handling while managing a connection pool could have easily taken an order of magnitude longer than this entire task took me (not including time spent downloading logs in the background). > Today, we have our choice of high-performance languages where it's easy to write, fast, safe code > In 1986, there would have been no comparable language Shell existed in 1986, and IIRC ftp could actually be scripted, a bit, so I think that it could have been used to run the download portion. Maybe I misremember? Lisp existed in 1986; the first edition of Common Lisp the Language was written in 1984. It certainly could have been used, and even in the 80s I think it would have good enough performance for log parsing. Certainly it would now, but that is after four decades of optimisation. His point about libraries, though, is excellent. The rise of open source libraries radically changed the software world for the better. I should note, though, that Plan 9 — which was developed in the late 80s — removed a ton of the accidental complexity in dealing with e.g. FTP servers. We could have much less accidental complexity, but we choose not to.

pydry 6y ago
>His point about libraries, though, is excellent. The rise of open source libraries radically changed the software world for the better. This was one of the first points made in no silver bullet (except he said buy not "download for free"). It was a lot less obvious in 1986 than it is today and arguably it's not even all that obvious today.
kazinator 6y ago
The reality is that Common Lisp would not fit onto a Unix workstation from 1986 in a light weight way, like firing off another /bin/sh or /bin/awk job. Machines had memories measured in single digit megabytes, often closer to zero than to 10.

karmakaze 6y ago

The fault in this discussion and what I was hoping to learn about is the reduction in complexity of evolving systems. This post and the original cited work is very much about one-shot waterfall development. This is not what is done today and has little bearing. How much of the effort in fitting a new requirement into an existing system is due to choices in the way it was decomposed and structured? I'm convinced there are much better than 2x better ways of keeping this effort low by choosing seams carefully and putting in the work when some are found to be in messy places. Another way to say this might be, any system could (for sake of argument) be built with 2x essential complexity, but an evolved system typically has a much higher factor due to its lineage. Finding ways of keeping this near 2x should be the focus.

taeric 6y ago

My gut is that the claim translates better to percentage of processing power per user. With such an excess of power today, you can scrape a ton of use out of it. That said, the use of scripting languages kind of proves the point. There is little attempt made to maximize compute. Only gets more true when you consider the absurd amount of memory being used.

pydry 6y ago

Fred Brooks suggests in 1986 that future productivity improvements focus more on: >Exploiting the mass market to avoid constructing what can be bought In 2020 danluu says that: >In 1986, there would have been no comparable language, but more importantly, I wouldn't have been able to trivially find, download, and compile the appropriate libraries and would've had to write all of the parsing code by hand, turning a task that took a few minutes into a task that I'd be lucky to get done in an hour. IME this demonstrates remarkable prescience by Brooks. The only real difference between Brooks' hypothetical future where software is assembled from bought components and the one danluu lives in is that the components are, in this case, free. Package managers might seem obvious and necessary today, but I don't think they were at all obvious in 1986 (or even in 1996). Fuck, golang didn't even think they were necessary in 2009. Also: >Looking at the specifics with the benefit of hindsight, we can see that Brooks' 1986 claim that we've basically captured all the productivity gains I don't believe that this was his claim. He claimed that productivity gains from improved programming language design would be incremental rather than game changing (hence: no silver bullet). Some people probably will argue that there is a 10x difference in language productivity due to language but I don't think that is the case. In fact, I think there's a class of people who keep getting proved wrong and wrong again about the relative importance of package management ecosystems over "game changing" language features like non nullable types that are incrementally better, but not an order of magnitude better.

ncmncm 6y ago
I see that you are unaware of the large collection of programming language improvements accumulated in recent decades, which I will guess is because your favored language mostly does not implement them.
- pydry 6y ago
  Which improvements caused a 2x+ productivity boost that my favored language does not have?
  - ncmncm 6y ago
    Guessing again based on your nick, improvements that draw on type theory or type mechanics.

BenoitEssiambre 6y ago

This is a pretty poor argument. This does not represent productivity in solving new problems, it just tells us it's worth reusing pieces for already solved problems. Yeah carpenters might seem very productive if they started with increasingly already built houses like programmers do. Solving the part of the problem that is novel still has irreducible complexity.

coldtea 6y ago
>This does not represent productivity in solving new problems, it just tells us it's worth reusing pieces for already solved problems. That's neither here nor there. First because most (if not all) of new problems can be solved by reusing existing parts (from languages and libs to widgets and cli tools). Second, because we tend to re-work on the same problems over and over for the most part, anyway, so improvements on those are still greatly important (even more so than novel problems actually, which are about potential uncertain future markets, whereas solved problems are what drive existing global markets, so dozens of trillions of dollars).

vannevar 6y ago

I don't think that Brooks was arguing that any given fixed programming task could not be made easier over time; clearly, even in 1986, there were programming tasks that had become trivial by use of existing code but were originally quite challenging to code on their own. I think Brooks was talking about the work of the individual programmer: given two identical programmers, with the same available libraries of existing code, is there a language or IDE that will make one 10x more productive than the other? And I think the answer is still, no.

brabel 6y ago
Exactly, this blog post missed the point completely. Unbelievable.
joshuafcole 6y ago
I think the lines between "available library of existing code", language, and IDE are much murkier than they might appear. If one programmer is given raw assembly and the other Python, you're probably going to see at _least_ a 10x productivity bump to the python user _even if they use no additional libraries_. Language and IDE choices impact the vocabulary available to express solutions in the same way.
- vannevar 6y ago
  You can certainly contrive a counter-example---even in 1986, you could say much the same thing about assembly and something like C or BASIC. But once that leap of abstraction was made, we seem to have hit a plateau of expressive power. Is the jump from C to Rust as empowering as the jump from assembly to C? I don't think it is.
jnwatson 6y ago
You're added premise is an implicit point of Luu's article. Or perhaps the point is that Brooks' framing doesn't really apply when you can avoid programming altogether. The silver bullet is open source. The ability to utilize the millions of hours of other people's work to accomplish your programming task is the game changer, and is what makes formerly complicated tasks trivial. The essential complexity of the software I deal with is about two orders of magnitude more complicated than stuff I dealt with 20 years ago. I'm not sure that 20 years from now it will be another two orders of magnitude, but perhaps with utility computing and higher level orchestration concepts, it very well might be.

ozim 6y ago

I have different idea about essential complexity and accidental complexity. I think examples in the article are all just accidental complexity. Essential complexity from how I read Fred Brooks is a domain knowledge like what an accountant does and what features he is using. All frameworks, servers and doing scp or queries in that is accidental complexity. Accountant does not need any React or Angular front end, they don't need a server. Where angular brings in reduction in accidental complexity since by using it I don't have to build a lot of stuff from scratch, it does not make tax laws any simpler. Where tax laws are essential complexity that needs to be coded in accounting software. The same with servers, going to cloud so reducing my accidental complexity by removing need to deal with servers is not making tax laws simpler or easier to grasp. This way in my opinion, essential complexity and accidental complexity holds really well as general idea.

sanxiyn 6y ago
Eh, you and Dan are in agreement? "This task ... is still nearly entirely accidental complexity", etc.
- ozim 6y ago
  Not really, because I think that tasks he picked as "essential complexity" are still fully "accidental complexity" in terms of how I understand Brooks writing. I cannot agree with Dan, because I feel that all "computer stuff" is accidental complexity in Brooks writing context. Some other replies change context into "from software dev perspective that is an essential complexity", now we are discussing what Fred Brooks wrote in 1987. If we change that context then I can agree with Dan, but if you want to refute argument you have to operate inside the same context. What I see that Dan argues, is accidental complexity is not getting easier throughout the years. He does not touch anything about essential complexity being well "complex" and that we cannot do much about it. So I don't agree that you can refute whole idea based on those examples.
  - sanxiyn 6y ago
    Where does Dan pick anything as essential complexity? Please cite quote. As far as I can tell, Dan considers everything in the article as accidential complexity, just like you do (and I do).

dustingetz 6y ago

Hypothesis: Constraints on time is the cause of accidental complexity. Given infinite time I would think longer, yak shave more, rewrite everything simpler, recur until perfect. In real life what constrains time? The business, its competition and available resources. Traded off vs speed of human thought and communication.

deleted 6y ago

[deleted]

heisenbit 6y ago

Brooks: > The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems. I think Brooks by focusing on the reality of the world reflected into topology of systems focuses on the essentials for the complexity. Yes, we can abstract some complexity but we also know these abstractions are leaky - heaping too many of them on top of each other and it topples. Even more important there are limits to how much most of us can juggle in their minds - exceed that threshold and we need a team with all the overhead that accompanies that. Once inter human communication is part of the process any optimization on tool level can only improve the situation by so much.

neolog 6y ago

Dan, I'm curious how all those people listed at the bottom read the post early. I sometimes want early feedback on my writing and you seem to be getting it from a lot of well known people. How do you do that? Do you just email them all and ask?

astine 6y ago

This essay badly misinterprets Brooks original essay. Brooks was talking specifically about the problem of building new systems from scratch.He was speaking about programming productivity. This essay is talking about operational tasks. Of course operational tasks can be speed up with improvements to software and hardware, that's the whole point of writing new software and developing new hardware. Furthermore, Brooks doesn't actually argue that there is no way to tackle essential complexity when writing new software. What he says is that there is no single technique that will speed up the process of writing code across the board, that is, there is no silver bullet. However, he does make allowance for techniques that allow us to break up complex systems into smaller systems which can then be solved independently. He specifically mentions object oriented programming as a tool which makes it easier for programmers to write software as components but CPAN would have been a better example. He just doesn't allow that OOP by itself would allow for an order of magnitute improvement (something which I think that experience has born out.) The author of this essay, points out a couple of operational tasks which involve programming were easier for them than they would have been 34 years ago because faster hardware noow exists and because software that already takes care of a subset of the problem now exists when it didn't before. Partially, this is confirmation of Brooks' idea that breaking down problems it the way to go long-term, but more critically, it's a failure to realize on the part of the author that they're speaking of productivity in their role as the user of a computer system whereas Brooks is speaking about productivity in designing new software products. edit: I think that the fundamental root of the essay's problem is that the author seems to have misssed what Brooks was referring to which he discussed accidental vs essential complexity. Brooks specifically pointed out that he was referencing Aristotle when he used those terms, and in this case essential doesn't mean unavoidable, but a part of the essential character of the problem itself. A more modern terminology would be intrinsic vs extrinsic. Complexity is extrinsic to programming if it comes from something other than the nature of the problem itself. If my computer is too slow to allow me to write a program, then that's probably not due the the nature of the program itself (except indirectly) and so represents a form of accidental complexity. Feature requirements however, represent essential complexity. You can manage essential complexity according to Brooks by dividing the problem up or avoiding altogether, but it's not something that will go away simply with better tooling or a better methodology, (unless that better tooling replaces software that you otherwise would have written.

zmmmmm 6y ago

I found the example given very contrived and the whole article slightly disingenuous because of it. Brooks was talking about building long-living applications serving complex business requirements. The article talks about a one off script with no ongoing life. It's not even really "software" at all in the terms Brooks was describing. It's like someone describing absolute limits on fuel efficiency for vehicles and then I get on a scooter and coast down a hill and jump off at the end so that it crashes into a wall and explodes, and using this example to claim they are wrong.

daotoad 6y ago
The difference between a one off script and a production-grade system with documentation, instrumentation, and monitoring is pretty big. Here's less than a minute's worth of questions I would want answered for a professional version of the log fetching example: * Is that list of a thousand servers static? * How is it updated? * What format is it in? * What do you do when the resource isn't available? * What other systems are impacted when the resource can't be contacted? * Is it acceptable to use a cached server list until it the resource can be updated? * How do you handle servers that don't respond when polled? If we retry connections, what scheme do we use? Linear or exponential back off? * After a service interruption, how do we resume data collection? Do we keep rolling forward or do we request older data? How quickly do we retrieve backed up data? Is service resumption plan different for the case when a single system is down vs when the log fetching system has an outage?
mchusma 6y ago
Agree. Nearly all the work of programming I have done is related to the task of trying to figure out the business problem in enough extreme detail to provide a solution. The main lever to improve productivity for me as a developer are open source tools, commercial APIs, and similar things. This isn't reducing the amount of programming, it's just moving it around. Magically 1,000x faster computers would maybe help me be 20% more productive.

0xbadcafebee 6y ago

Let's say you're building a home. Foundation is poured and it's time to frame it. If you hire some random guys with power tools standing outside a home depot, it'll take weeks to months for them to finish, and might well end up crappy. But hire two 55-year-old men with hammers, nails and a single skil-saw, and they can have it up in 3-4 days. Or grab your Amish neighbors and raise a barn in a day. Productivity is not bounded by your access to technology, it's bounded by the problem you need to solve and the means you use to solve it. > Brooks explicitly dismisses increased computational power as something that will not improve productivity ("Well, how many MIPS can one use fruitfully?", more on this later), but both storage and CPU power (not to mention network speed and RAM) were sources of accidental complexity so large that they bounded the space of problems Brooks was able to conceive of. Yet here in the future, our excess of storage, CPU power and network bandwidth have bounded our ability to be fruitful with them. If you add up the MIPS we use today and compare it to what solutions we're implementing with it, we are vastly less productive today. Brooks was right, but in a weird way. There is a bound on programmer productivity, and it is the programmer. In the past, people wouldn't have set themselves up to need to collect a billion billion logs to analyze some tiny metric among them. Anyone would've seen that as poor planning and a horrible waste of resources. Why would you bury needles in billions of haystacks, and then need to hastily build a robot that finds needles? Why not just keep the needles in a pile of needles? Do you really even need all those needles? > Thinking that the tools you personally use are as good as it gets is an easy trap to fall into. This actually seems like a hard trap for most to fall into. Nearly every software engineer I've met either thinks they could make a better tool, or just wants to try for the hell of it. On the other hand, most engineers think that building a better mouse trap will solve all their mouse problems.

ncmncm 6y ago

The only way Brooks's argument makes sense is that we naturally tackle problems as big as the resources available to us permit. So, our daily load of programming hardly changes, because as we get better equipped, more is asked for. But in that interpretation, none of his supporting detail applies.

sontek 6y ago

Maybe off topic but this site might have convinced me to move to Firefox. Are all of you reading this page on mobile or a small screen? I have a large monitor and this page expands full width of the screen and the lines are extremely hard to follow. It turns out Firefox has a "Reader View" that makes sites like this be put into a nice readable column. Here is a comparison of the two: https://i.imgur.com/plV0e7l.png https://i.imgur.com/plV0e7l.png I find it interesting that this post reached to the top of HN with such an unreadable layout and so I was interested in if everyone has some tricks for reading content like this or is Firefox/Reader Mode the best way?

konjin 6y ago

Going to hard disagree. What he's saying there is that today we have free implementations of libraries which solve incidental complexity and those libraries work (nearly) everywhere (nearly) the same. I worked in finance on K and the name makes in impossible to use google/stack overflow to find anything about the language. Which meant I had a paid for help and tech support from a company that's been largely unchanged from the 80s. It was fucking phenomenal. Once I gave up the idea that I knew best and started working in the way the language expected me my productivity was an order of magnitude higher than it would have been in a cobbled together python pipeline like the one he uses there. And because someone was responsible for the language and tooling I could use scripts from before I was in primary school, and they worked as expected. I dare you to try and do the same with any language that isn't C.