5 ms·

I have different idea about essential complexity and accidental complexity. I think examples in the article are all just accidental complexity. Essential compl

by ozim 6y ago

I have different idea about essential complexity and accidental complexity. I think examples in the article are all just accidental complexity. Essential complexity from how I read Fred Brooks is a domain knowledge like what an accountant does and what features he is using. All frameworks, servers and doing scp or queries in that is accidental complexity. Accountant does not need any React or Angular front end, they don't need a server. Where angular brings in reduction in accidental complexity since by using it I don't have to build a lot of stuff from scratch, it does not make tax laws any simpler. Where tax laws are essential complexity that needs to be coded in accounting software. The same with servers, going to cloud so reducing my accidental complexity by removing need to deal with servers is not making tax laws simpler or easier to grasp. This way in my opinion, essential complexity and accidental complexity holds really well as general idea.

sanxiyn 6y ago
Eh, you and Dan are in agreement? "This task ... is still nearly entirely accidental complexity", etc.
- ozim 6y ago
  Not really, because I think that tasks he picked as "essential complexity" are still fully "accidental complexity" in terms of how I understand Brooks writing. I cannot agree with Dan, because I feel that all "computer stuff" is accidental complexity in Brooks writing context. Some other replies change context into "from software dev perspective that is an essential complexity", now we are discussing what Fred Brooks wrote in 1987. If we change that context then I can agree with Dan, but if you want to refute argument you have to operate inside the same context. What I see that Dan argues, is accidental complexity is not getting easier throughout the years. He does not touch anything about essential complexity being well "complex" and that we cannot do much about it. So I don't agree that you can refute whole idea based on those examples.
  - sanxiyn 6y ago
    Where does Dan pick anything as essential complexity? Please cite quote. As far as I can tell, Dan considers everything in the article as accidential complexity, just like you do (and I do).
    - burakemir 6y ago
      "Let's see how this essential complexity claim holds for a couple of things I did recently at work:" "For some queries, it's arguably zero — my work was necessary only because of some arbitrary quirk and there would be no work to do without the quirk. But even in cases where some kind of query seems necessary, I think it's unbelievable that essential complexity could have been more than 1% of the complexity I had to deal with." It's very clear that the author of the post understands "essential complexity" as related to thinking about the query they need to write (data structures). The issue is that in today's computing environments application programming is only a fraction of solving the problem of delivering the application. They too see the essential (or inherent) complexity as being a domain and requirements related thing, but they draw the conclusion that they are somehow not part of it. It is possible the whole app and system would be unnecessary if the essential complexity was addressed differently. And it is also possible that due to the current way of doing things, there are new sources of essential (inherent) complexity. Brooks couldn't have foreseen productivity gains coming from the internet and collaboration. But if you factor all that in and create a new baseline, faster machines still don't tell you what is the best way to deal with changes in requirements or how to represent a postal address so that it works internationally.
unabst 6y ago
In your example, the tax code is simply the abstraction layer one level down that you have no control over. So to translate your example, HTML specs would be essential complexity and all authored HTML would be accidental complexity to a front-end web programmer. But this makes complexity a matter of context. A browser is built using a programming language and uses interfaces provided by the OS and other libraries. This makes the browser accidental. And you can take this all the way down. Assembly is accidental with respect to the processor being essential. But now we see how complexity flows through each layer. Just as the accidental complexity of the tax code becomes essential complexity for the accountant, the accidental complexity of HTML becomes essential complexity for the web programmer. And a popular pass time is complaining how much simpler everything could be, only if. (not sure if this is what the original article had in mind but this model seems accurate)
- reificator 6y ago
  > So to translate your example, HTML specs would be essential complexity and all authored HTML would be accidental complexity to a front-end web programmer. The HTML specs are not essential complexity, are you joking? Accounting software that handles tax codes is the essential piece. HTML is just one approach to building an application. Even though I whip out web dev quite often, I find it scary that more and more people view it as the only option or the default approach. To say it again, the HTML specification is not essential complexity in the context of an accounting app.
  - unabst 6y ago
    You're missing the point which is context. If you're the accountant, the tax code is essential. HTML is accidental. If you're the programmer, then HTML is essential. The outcome of your page is accidental. Accidental is what you have control over. Essensial is what you're forced to work with. If you're being forced to code in HTML, just as an accountant is forced to work with the tax code, the domain knowledge as provided and exposed as specs, be it HTML specs or the Tax Code cannot be altered. So if the argument is that the tax code is essential complexity for an accountant, then the analogy holds. To an accountant writing an HTML page for their firm or a one page app implementing some accounting feature for clients, HTML is all accidental. They could use anything else. What is essential to the accountant is accounting. But since we're talking programming, the situation applied to a programmer: The HTML specs are essential complexity for a web programmer. They must work with it, and cannot work around it. And this is how complexity is passed on through the layers of the system. And with the case of an accountant needing programming, the line of complexity between them is violated all the time. A web programmer hired by an accountant fails to implement good accounting software by being bound by the essential complexities of HTML. The accountant doesn't care what cannot be done because of some HTML spec. The web programmer must find ways to work around this through engineering, or compromise. This leads to further accidental complexity. edit/addendum: > I whip out web dev The moment you go with web dev, this is the act that now makes HTML specs essential. The only way to escape HTML specs is to use something else, which you cannot do if you're already decided on a web app. Just as the only way an accountant can escape the tax code would be to change professions. To align the argument with the original article, if HTML specs are half the web app, then the web app cannot be less complex than, say, half. And the only way to get there is to delete everything you did/added which is your page/program. Coding complexity can never break the essential complexity barrier, and accidental complexity in the form of your code is always a positive value. edit/addendum2: When adding a framework or library to reduce complexity, there must be a pragmatic replacement/swap of original essential complexity abstractions. The new library is positive, in that you're adding more code, and carries its essential complexity. But when the complexity of, say, jQuery can replace a priori javascript essential complexity, then we can make it out ahead. With the added complexity of replacing complexity.
archibaldJ 6y ago
Agree. Factoring in the business process (esp a somewhat formalized one) as well as the notion of economic value would help to introduce a pragmatic separation between essential and accidental complexity. I guess what gets interesting is when the business process is fussy itself (e.g. in a start-up setting) and that is when the direction of programming and software development at large takes in a more reinforcement learning discourse v.s. a pure top-down reductionistic one. Perchance that is also when the categorisation of essential vs accidental complexity becomes more ambiguous in the human process.
mc10 6y ago
I think this assumes a fine distinction between "domain" and "non-domain" knowledge that often doesn't exist. If you're writing a trading strategy, the technical details of how the code's executed often dictate what kind of trading you're planning on doing. e.g. How do I balance speed vs how sophisticated the algorithm is? Also, if your "domain" is technical in nature (e.g. building a new database), it's not clear at all what parts are "essential" and what parts are "accidental". Is the file system accidental? Is the underlying data store (SSDs, spinning disks) accidental? Often the purpose of the project is intrinsically tied to the technical capabilities of the platform it's built on top of.
pm 6y ago
I personally refer to these concepts respectively as inherent and incidental complexity, what should be classed as inherent complexity is anything that directly supports the description of the problem. Everything else is incidental complexity (and we mostly write incidental complexity). The big thing that I don't hear often is about where complexity lives. Complexity is like a pile of receipts you need to hunt down to do your tax - you can have it distributed all over your house, you can sweep up into a pile, or you could file it. The first is by far the most complex, and I've worked on codebases that looked like that. Even the most trivial of changes takes a long time.
- jariel 6y ago
  I think the bifurcation might be more clearly described as 'problem space' and 'solution space' complexity. The 'complexity of the problem' is inherent, by definition. Unless accountants want to literally change the requirements. The complexity of the solution space is something else entirely and it may be hard to derive theoretically ... But with sufficient tooling, such as AI, maybe it becomes trivial. If we have 'AI Apis' that can immediately handle complex problems and dissolve them into something really simple, then many apparently complex domains could be reduced to something simple. As long as we're allowed to use 'other software' and not have it 'count' as part of the complexity we are measuring, well, the then complex of the 'app' tends towards 0. You can already see this a little bit with CRUD apps, which often feel like just gluing some forms together.
megameter 6y ago
From a philosophical standpoint, the starting line for "complex" is in something you take it upon yourself to pursue. The world we inhabit has a complex ecosystem, physics, etc; our study of it is far from complete. But every time we learn something about it a technical advance becomes possible. Nobody has to keep the gears turning in this vast society. But someone at some point decided it was worth it to create those gears, without answering all the how's and why's: basic science doesn't tell us how to engineer things, only that we have ways of dong so. So the "accident" of it emerges like a freeway pileup, from different domains of study colliding. And with computing being the workhorse technology for everything now, the crashes are so frequent as to be unremarkable.

by ozim 6y ago

sanxiyn 6y ago

Eh, you and Dan are in agreement? "This task ... is still nearly entirely accidental complexity", etc.

ozim 6y ago
Not really, because I think that tasks he picked as "essential complexity" are still fully "accidental complexity" in terms of how I understand Brooks writing. I cannot agree with Dan, because I feel that all "computer stuff" is accidental complexity in Brooks writing context. Some other replies change context into "from software dev perspective that is an essential complexity", now we are discussing what Fred Brooks wrote in 1987. If we change that context then I can agree with Dan, but if you want to refute argument you have to operate inside the same context. What I see that Dan argues, is accidental complexity is not getting easier throughout the years. He does not touch anything about essential complexity being well "complex" and that we cannot do much about it. So I don't agree that you can refute whole idea based on those examples.
- sanxiyn 6y ago
  Where does Dan pick anything as essential complexity? Please cite quote. As far as I can tell, Dan considers everything in the article as accidential complexity, just like you do (and I do).
  - burakemir 6y ago
    "Let's see how this essential complexity claim holds for a couple of things I did recently at work:" "For some queries, it's arguably zero — my work was necessary only because of some arbitrary quirk and there would be no work to do without the quirk. But even in cases where some kind of query seems necessary, I think it's unbelievable that essential complexity could have been more than 1% of the complexity I had to deal with." It's very clear that the author of the post understands "essential complexity" as related to thinking about the query they need to write (data structures). The issue is that in today's computing environments application programming is only a fraction of solving the problem of delivering the application. They too see the essential (or inherent) complexity as being a domain and requirements related thing, but they draw the conclusion that they are somehow not part of it. It is possible the whole app and system would be unnecessary if the essential complexity was addressed differently. And it is also possible that due to the current way of doing things, there are new sources of essential (inherent) complexity. Brooks couldn't have foreseen productivity gains coming from the internet and collaboration. But if you factor all that in and create a new baseline, faster machines still don't tell you what is the best way to deal with changes in requirements or how to represent a postal address so that it works internationally.

unabst 6y ago

In your example, the tax code is simply the abstraction layer one level down that you have no control over. So to translate your example, HTML specs would be essential complexity and all authored HTML would be accidental complexity to a front-end web programmer. But this makes complexity a matter of context. A browser is built using a programming language and uses interfaces provided by the OS and other libraries. This makes the browser accidental. And you can take this all the way down. Assembly is accidental with respect to the processor being essential. But now we see how complexity flows through each layer. Just as the accidental complexity of the tax code becomes essential complexity for the accountant, the accidental complexity of HTML becomes essential complexity for the web programmer. And a popular pass time is complaining how much simpler everything could be, only if. (not sure if this is what the original article had in mind but this model seems accurate)

reificator 6y ago
> So to translate your example, HTML specs would be essential complexity and all authored HTML would be accidental complexity to a front-end web programmer. The HTML specs are not essential complexity, are you joking? Accounting software that handles tax codes is the essential piece. HTML is just one approach to building an application. Even though I whip out web dev quite often, I find it scary that more and more people view it as the only option or the default approach. To say it again, the HTML specification is not essential complexity in the context of an accounting app.
- unabst 6y ago
  You're missing the point which is context. If you're the accountant, the tax code is essential. HTML is accidental. If you're the programmer, then HTML is essential. The outcome of your page is accidental. Accidental is what you have control over. Essensial is what you're forced to work with. If you're being forced to code in HTML, just as an accountant is forced to work with the tax code, the domain knowledge as provided and exposed as specs, be it HTML specs or the Tax Code cannot be altered. So if the argument is that the tax code is essential complexity for an accountant, then the analogy holds. To an accountant writing an HTML page for their firm or a one page app implementing some accounting feature for clients, HTML is all accidental. They could use anything else. What is essential to the accountant is accounting. But since we're talking programming, the situation applied to a programmer: The HTML specs are essential complexity for a web programmer. They must work with it, and cannot work around it. And this is how complexity is passed on through the layers of the system. And with the case of an accountant needing programming, the line of complexity between them is violated all the time. A web programmer hired by an accountant fails to implement good accounting software by being bound by the essential complexities of HTML. The accountant doesn't care what cannot be done because of some HTML spec. The web programmer must find ways to work around this through engineering, or compromise. This leads to further accidental complexity. edit/addendum: > I whip out web dev The moment you go with web dev, this is the act that now makes HTML specs essential. The only way to escape HTML specs is to use something else, which you cannot do if you're already decided on a web app. Just as the only way an accountant can escape the tax code would be to change professions. To align the argument with the original article, if HTML specs are half the web app, then the web app cannot be less complex than, say, half. And the only way to get there is to delete everything you did/added which is your page/program. Coding complexity can never break the essential complexity barrier, and accidental complexity in the form of your code is always a positive value. edit/addendum2: When adding a framework or library to reduce complexity, there must be a pragmatic replacement/swap of original essential complexity abstractions. The new library is positive, in that you're adding more code, and carries its essential complexity. But when the complexity of, say, jQuery can replace a priori javascript essential complexity, then we can make it out ahead. With the added complexity of replacing complexity.

archibaldJ 6y ago

Agree. Factoring in the business process (esp a somewhat formalized one) as well as the notion of economic value would help to introduce a pragmatic separation between essential and accidental complexity. I guess what gets interesting is when the business process is fussy itself (e.g. in a start-up setting) and that is when the direction of programming and software development at large takes in a more reinforcement learning discourse v.s. a pure top-down reductionistic one. Perchance that is also when the categorisation of essential vs accidental complexity becomes more ambiguous in the human process.

mc10 6y ago

I think this assumes a fine distinction between "domain" and "non-domain" knowledge that often doesn't exist. If you're writing a trading strategy, the technical details of how the code's executed often dictate what kind of trading you're planning on doing. e.g. How do I balance speed vs how sophisticated the algorithm is? Also, if your "domain" is technical in nature (e.g. building a new database), it's not clear at all what parts are "essential" and what parts are "accidental". Is the file system accidental? Is the underlying data store (SSDs, spinning disks) accidental? Often the purpose of the project is intrinsically tied to the technical capabilities of the platform it's built on top of.

pm 6y ago

I personally refer to these concepts respectively as inherent and incidental complexity, what should be classed as inherent complexity is anything that directly supports the description of the problem. Everything else is incidental complexity (and we mostly write incidental complexity). The big thing that I don't hear often is about where complexity lives. Complexity is like a pile of receipts you need to hunt down to do your tax - you can have it distributed all over your house, you can sweep up into a pile, or you could file it. The first is by far the most complex, and I've worked on codebases that looked like that. Even the most trivial of changes takes a long time.

jariel 6y ago
I think the bifurcation might be more clearly described as 'problem space' and 'solution space' complexity. The 'complexity of the problem' is inherent, by definition. Unless accountants want to literally change the requirements. The complexity of the solution space is something else entirely and it may be hard to derive theoretically ... But with sufficient tooling, such as AI, maybe it becomes trivial. If we have 'AI Apis' that can immediately handle complex problems and dissolve them into something really simple, then many apparently complex domains could be reduced to something simple. As long as we're allowed to use 'other software' and not have it 'count' as part of the complexity we are measuring, well, the then complex of the 'app' tends towards 0. You can already see this a little bit with CRUD apps, which often feel like just gluing some forms together.

megameter 6y ago

From a philosophical standpoint, the starting line for "complex" is in something you take it upon yourself to pursue. The world we inhabit has a complex ecosystem, physics, etc; our study of it is far from complete. But every time we learn something about it a technical advance becomes possible. Nobody has to keep the gears turning in this vast society. But someone at some point decided it was worth it to create those gears, without answering all the how's and why's: basic science doesn't tell us how to engineer things, only that we have ways of dong so. So the "accident" of it emerges like a freeway pileup, from different domains of study colliding. And with computing being the workhorse technology for everything now, the crashes are so frequent as to be unremarkable.