9 ms·

The growing complexity of modern software systems

133 points by tut-urut-utut 5y ago

greenail 5y ago
One thing I've seen is that forced prioritization can help reduce complexity. The idea is that if you don't have enough resources to re-invent the wheel you will be forced to think harder about what you choose to work on. This has to be cultural and supported all the way to the top of the organization. Counterintuitively too many resources can slow you down.
- StephenJGL 5y ago
  This is true. Someone has to be the architecture person who can separate the actual complexity created by your requirements as they exist from manufactured complexity created by using architecture techniques that don't follow from requirements.
newaccount2021 5y ago
aka the most dangerous programmers are at the "intermediate" skill level beginning coders do not understand how to create complexity or are just afraid of the pointy tools expert coders don't need to flex, understand the downsides of more code outweigh the upsides it is the intermediate developer who will pave the road to hell with "nice to have", "patterns", "abstractions"...
bedobi 5y ago
I'm not sure I agree with this. In the past, software development and how things were deployed etc was the wild west. Every company and team had completely different standards and methods etc and lots of it was manual and little to no testing etc etc. Now, with things like AWS, Terraform, increasingly converging popular programming languages, frameworks and best practices like tests, code review etc, in some ways things are less complex.
- StephenJGL 5y ago
  I agree with your disagreement. I've been in involved in web dev for 30 years. There is more moving parts, but IMO they better align with the actual complexity. This is likely only true for professional, production grade apps where the full range of concerns are in play such as security, maintenance, testing, migrations, scaling, etc. Deploying a simple app is more complex. For the apps I built I don't have to plan server purchases months out. I don't have to coordinate multiple apps on one machine. I don't have to worry about swapping out LBs and Firewalls as a nightmare connectivity issue. I don't have to scale file storage regularly. I can scale up service DBs as needed. All of that reduces total complexity. Beyond that, in some instances even for simple apps there is less complexity once you know the technology. That's not complexity though. That's a buffet of useful options. Good architecture decision are even more critical today.
  - deltaonefour 5y ago
    > I don't have to worry about swapping out LBs and Firewalls as a nightmare connectivity issue. I don't have to scale file storage regularly. I can scale up service DBs as needed. All of that reduces total complexity. You're wrong. Complexity has increased by a huge magnitude so much so that it's actually unmanageable by a single person. What's going on in your case is that the complexity of all of this has been abstracted away from you by amazon. They have entire teams of people making sure you don't have to deal with the actual complexity. That's the difference. This isn't an actual abstraction in the same way a high level language abstracts the details away from you. This is an abstraction where a team of people take care of you. It is not a true abstraction because you cannot wield this abstraction yourself in the same way you can wield a high level programming language. Make no mistake, the actual complexity of it all has increased by huge orders of magnitude.
    - squabble 5y ago
      I think they mean that it reduces the complexity that a single developer has to deal with.
zwieback 5y ago
A lot of nostalgia for an era that didn't exist: “It has never been more difficult to be a software developer than it is today...” and “Everything was so much simpler before..." SW dev has always been overly complex, long before AWS and the web existed. It's just how we do things in software engineering. Too many choices and lack of physical constraints leads us right to the edge of where things totally fall apart.
- bmitc 5y ago
  Just because it's always been complex doesn't mean the complexity isn't increasing. Software development is ridiculously complex, increasingly so, and one often doesn't have any tools to deal with it.
  - adminscoffee 5y ago
    i think it's done for job security. kinda like antivirus software always had new "threats", but that's probably not true, it just feels sometimes like we are reinventing the wheel to keep tech jobs relevant, at least in certain fields
    - qsort 5y ago
      If this were the case, you'd see at least some people/orgs doing better and spending less than the current "best practice". I don't think it's reasonable to say software developers are just fooling everyone.
- pjmlp 5y ago
  VB 6/Delphi in a 3-tier application versus HTML/CSS/JS npm spaghetti pretending to be a GUI framework instead of interactive documents.
  - deleted 5y ago
    [deleted]
redact207 5y ago
I don't quite agree with much of this article. It paints the industry with a broad brush and not everyone has had the same experience - least of which me. The article starts: > The shift from building applications in a monolithic architecture hosted on a server you could go and touch, to breaking them down into multiple microservices, packaged up into containers, orchestrated with Kubernetes, and hosted in a distributed cloud environment, marks a clear jump in the level of complexity of our software. Add to that expectations of feature-rich, consumer-grade experiences, which are secure and resilient by design, and never has more been asked of developers. Comparing the most complicated way to build and run an app with something that isn't as complicated is just a straw man. Today you can still build a simple, monolithic app and deploy it into a completely managed service. All the points raised address certain problems: - microservices - sliced out of a monolith when complexity or scalability becomes difficult - containerisation - deploy an app with all of its dependencies down right down to the OS - orchestration - ensure reliability when pods/tasks fail, rolling updates, commoditize hardware - distributed cloud environment - remove the need for on-prem, or have the ability to elastically scale without monthly VPS commitments - secure & resilient by design - apps that make money should have good uptime, not leak data None of those are mandatory for all apps - it's pick and choose as and when you get the need. But also what is the alternative provided from "years ago" when things were seemingly simpler? - microservices - ever tried keeping velocity when building XMLSOAP based apps? - containerisation - write a bunch of ansible scripts or maintain disk images that can be sent to server fleets with bash scripts, else more scripts to apt-get/apk install stuff. On windows before package managers? enjoy shipping zips and exes with your code. - orchestration - scripts? - distributed cloud environment - build your own datacentre if you're big enough, lease out some racks if your not and have a systems team maintain them, patch them, RAID stripe disks and swap out dying infrastructure; or sign up to managed VPSs and raise a requisition order each time you want a new server that'll take a week to go through. - secure & resilient by design - a lot to do for a 6-9s uptime; so just lock down your release process with a ton of bureaucracy and approval gates by different teams. not releasing is great to improve uptime SLAs. I don't have a good impression of the "good old days". The complexity was always there, it was just managed by different teams. The real question is should a software developer need to know and work across all of these things, or are people who no longer have "systems management" job move into DevSecOps and augment a software team?
ogazitt 5y ago
Every few years, some version of this article comes out. The version of 30 years ago lamented how programmers don't really understand the hardware anymore because a compiler abstracts the machine. 20 years ago, it was about how virtual machines (like the JVM / CLR) abstract memory management and developers write bloated software. We simply have more layers of abstraction that we depend on, and the interesting problems move up the stack. That trend will continue until the machines take over and write software. Until then, we'll continue to see this article show up, s/kubernetes/next-new-hotness.
- erik_seaberg 5y ago
  The stack also improves. Early versions of the JVM were very noticeably slower than C++, and the first CLR GC was so poor that my team at Microsoft had to resort to arrays of structs for keeping a service’s state. These days they’re great, and it’s hard to do better without putting a lot more effort into something like Rust.
  - adrianN 5y ago
    The JVM is great, but I think it's not too difficult to do better in a language that doesn't force so much pointer-chasing. Cultural differences probably also play a role. Java systems tend to be engineered such that more pointer chasing happens even in places where the language doesn't force it.
    - WalterGR 5y ago
      Does the CLR improve on that much, with its stack-allocated structs?
  - jcelerier 5y ago
    > Early versions of the JVM were very noticeably slower than C++, The fastest java software I use feels much slower than the slowest c++ one (talking about UI lag)
    - 5y ago
rglover 5y ago
Microservices and serverless were the "we've jumped the shark" point for me. Both can be a great pattern applied properly (nasty scaling problems), but they became a religion or a replacement for entire stacks. I see so many devs these days who are clearly confused but have no idea what they're doing is overly complex—they just think it's normal. AWS and the like are great, but the move to make everything cloud-based is going to lead to a lot of unmaintainable (or in extreme cases, unsalvageable) messes later. It will be a sad day when devs only understand how to build or deploy something if it's using a heavily-abstracted corpo stack, or even worse, "no code."
- adminscoffee 5y ago
  no code will increase alongside ai, developers won't have jobs in the future. it'll be interfaces where the system builds it's solution okay after you tell it what you want. basically jit on steroids
  - bruce343434 5y ago
    all of the no code creations I have seen posted on this site in the past month are incredibly slow and sluggish. Sometimes even straight up buggy, for instance that font compare picker thing, would sometimes load no font (null) on the left side. As a paradigm, it has a long road to go. If people really think that no code is acceptable, then I am going to cry. It won't be an improvement.
    - rglover 5y ago
      Yes. It's not about improvement, it's about ease of use and reduction of work, or: convenience. That convenience is going to bite an incredible number of people from behind and send a lot of businesses spiraling.
  - Ekaros 5y ago
    Will there really be a no code, or is that just moving the level of abstraction one more level higher... While still needing occasional coding to solve those bits that aren't available...
deleted 5y ago
[deleted]
blondin 5y ago
modern software has become complex. but people, especially those involved in this complexity, insist that this is all normal. simplicity (or whatever it will be) is coming for modern software. we are seeing the early stages of that upcoming change: denial & resistance. denial is when we hear engineers say that software is simpler or no more complex than it was. resistance is when engineers try to convince us that "we need that complexity". following the well-known psychological curve, next up is acceptance. and i have started to see it already. kubernetes, for example, continue to deprecate poorly designed or overly complex features. it is hard because of backward compatibility. all that to say that change is coming. maybe we will simplify and start the cycle all over again :)
- systemvoltage 5y ago
  I am not a fan of denial & resistance arguments. Do bad thing X. Try to convince others that X is good and its failure of acceptance is just people's denial. Next step is the well-known psychological curve that you will accept X eventually. This can be applied indiscriminately to anything one wants to push without having to defend the specifics. We should instead question all assertions - there is no law out there that denial always precedes acceptance.
  - CRConrad 5y ago
    > there is no law out there that denial always precedes acceptance. Yeah. "Denial, resistance, acceptance" -- isn't that from the six (or was it seven?) stages of grief? But grief is for something inevitable and irresistible, a fait accompli: When a loved one has died that has already happened and there is nothing you can do about it. Here, though, if enough of us think that our industry is going the wrong way, maybe we can still stop it.
kashyapc 5y ago
For what it's worth, one of the QEMU (the open source emulator and virtualizer) maintainers, Paolo Bonzini, recently gave a talk about this topic. Here's an extended summary of that on LWN: "A QEMU case study in grappling with software complexity" — https://lwn.net/Articles/872321/ https://lwn.net/Articles/872321/ Although the article uses QEMU as an example, the lessons discussed in it are equally applicable to many other projects. (If you're short on time, read the intro, "Sources of complexity", "Ways to fight back", and the short conclusion. The QEMU command-line case study section might be interesting for those who use a lot of QEMU, or to those devs who use QEMU to write higher level tools.)
cjfd 5y ago
The biggest problem seem to be microservices. Basically, an army of cargo culters missed the small print that you should not even be thinking about them unless you work with at least 100 programmers. The talking heads that popularized microservices are also guilty of this. They occasionally mention that microservices are only for large teams but not nearly often enough.
- comeonseriously 5y ago
  Cargo culting is just what you have to learn to accept and work around, because there will always be developers who fall for it. A decade ago, everyone was writing dependency injection frameworks because that was the hotness and debugging became a nightmare, but, hey!, we completely decouple EVERYTHING! Then there was no sql (mongo etc) and if you were still using sql, your system was shit. Oh and way before that there was GoF and you ended up with UserBillingFacadeFactoryFactory abstractions. None of these technologies are bad. They're just bad in the hands of mediocre developers.
- packetlost 5y ago
  Yes and no. There are legitimate reasons to build microservices with small teams (such as requiring differing runtime environments due to hardware/architectural constraints). They should never be the default at a team below that scale though.
davidgl 5y ago
Is it more complicated though ? I've been developing for over 30 years, and I don't feel like its got any worse, I think it's the nature of the medium. These days I fight with npm or nuget dependencies problems, 20 years ago I was trying to write Delphi code to do simple REST requests, the complexity moves around but is still there. Out of the Tar Pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf) was written 15 years ago, and it's all about curbing complexity
- ratww 5y ago
  I think this is an example of simple vs easy. Writing request code 20 years ago was harder, because you needed to read docs and write a lot by hand. But it was simpler because it didn't add any additional hidden complexity from libraries or whatnot. At least in most languages it got better: we only need something simple and easy, like Javascript's fetch. Does complexity we don't have to maintain count? About Out of the Tar Pit, complexity manifests itself in various shapes. The essay itself lists some that have gotten better: state is one, with people using more functional code. The "Power corrupts" part has also gotten better, IMO, with languages becoming simpler (Go) or more restrictive (Rust). I think the "Code Volume" part is still a big one. One thing I'd like to see curbed to avoid code volume are the proliferation of code for handling edge cases. Having libraries and code able to handle tens of edge cases is both a blessing a curse. However most developers and business people can only see the blessing part.
lbriner 5y ago
As other posters have said, I think there is somewhat too much nostalgia about how "easy" it was in the old days. I was reading about software dev in the 70s and it took like 4 years to write what we could do in a day today. Also, I think that relatively, we are still in the early days of software where lots of people are clamouring to invent the silver bullet to make it all work better, but after a bit longer, we will have more experience of what works and doesn't work and we will consign certain things to history. Think about what medicine was like in the first few centuries? Leeches, trepanning, toxic "medicines", they presumably had the same issues until they established a much better foundation of what works and what doesn't which has made medicine move on. We will keep some of our cool things but others (XML anyone!) will just die off as we realise they are a solution to a problem that most of us don't have.
- v-erne 5y ago
  >>Think about what medicine was like in the first few centuries ... I spend thinking about medicine quite a lot (health problems in my family run deep) and this does not give me great deal of hope for our industry at all. There is so much quackery everywhere - most of diagnostics is based on some magic guess work. If symptoms are non-specific than You are out of luck - You have to spend years visiting random doctors hoping that one of them will have fresh enough memory of some book that he read years ago to guess right. And even when the diagnosis is easy the therapy can be not that far away from leeches. Few years ago I had wart on my hand and dermatologist offered me three options: - freeze it to death - burn it to death with acid - cut it out If You look at this options closely they require some high tech to be possible but from medical point of view they are as primitive as leeches. I honestly belive that in 100 years time when nanorobots will be available to everyone the doctor will prescrabie therapy that is basically nanorobots attacking the wart with little flamethrowers.
- fn1 5y ago
  > We are still in the early days of software where lots of people are clamouring to invent the silver bullet to make it all work better [...] I have a feeling that (especially in devOps/kubernetes) often there is a wish to have a magic button/command that does everything. Like in a rube-goldberg machine. What is forgotten then is the cognitive overhead of templating YAML inside templating YAML which is needed for the magic-button to be configured. And if the buttons behaviour needs to be changed very few weeks, not much is gained.
sswastioyono18 5y ago
some mention it about microservices but no one mention about the emerge of cloud? when we are still using monolith and using old deploy to one server, that would be easy. Then there is scaling issue and that's where the cloud came because really, when you have to manage a lot of things at once, it's better to let someone manage it for you and you can focus on your business logic. Obviously, the practice wasn't perfect. A lot of "micronization" comes up instead domain driven problem but that's the problem on mostly people not the tech
aryehof 5y ago
Perhaps this suits todays technology focused developer. There has been no progress on how to effectively determine and model complex business domain requirements into code, so just focus on the technological requirements and plumbing.

133 points by tut-urut-utut 5y ago

One thing I've seen is that forced prioritization can help reduce complexity. The idea is that if you don't have enough resources to re-invent the wheel you will be forced to think harder about what you choose to work on. This has to be cultural and supported all the way to the top of the organization. Counterintuitively too many resources can slow you down.

StephenJGL 5y ago
This is true. Someone has to be the architecture person who can separate the actual complexity created by your requirements as they exist from manufactured complexity created by using architecture techniques that don't follow from requirements.

newaccount2021 5y ago

aka the most dangerous programmers are at the "intermediate" skill level beginning coders do not understand how to create complexity or are just afraid of the pointy tools expert coders don't need to flex, understand the downsides of more code outweigh the upsides it is the intermediate developer who will pave the road to hell with "nice to have", "patterns", "abstractions"...

bedobi 5y ago

I'm not sure I agree with this. In the past, software development and how things were deployed etc was the wild west. Every company and team had completely different standards and methods etc and lots of it was manual and little to no testing etc etc. Now, with things like AWS, Terraform, increasingly converging popular programming languages, frameworks and best practices like tests, code review etc, in some ways things are less complex.

StephenJGL 5y ago
I agree with your disagreement. I've been in involved in web dev for 30 years. There is more moving parts, but IMO they better align with the actual complexity. This is likely only true for professional, production grade apps where the full range of concerns are in play such as security, maintenance, testing, migrations, scaling, etc. Deploying a simple app is more complex. For the apps I built I don't have to plan server purchases months out. I don't have to coordinate multiple apps on one machine. I don't have to worry about swapping out LBs and Firewalls as a nightmare connectivity issue. I don't have to scale file storage regularly. I can scale up service DBs as needed. All of that reduces total complexity. Beyond that, in some instances even for simple apps there is less complexity once you know the technology. That's not complexity though. That's a buffet of useful options. Good architecture decision are even more critical today.
- deltaonefour 5y ago
  > I don't have to worry about swapping out LBs and Firewalls as a nightmare connectivity issue. I don't have to scale file storage regularly. I can scale up service DBs as needed. All of that reduces total complexity. You're wrong. Complexity has increased by a huge magnitude so much so that it's actually unmanageable by a single person. What's going on in your case is that the complexity of all of this has been abstracted away from you by amazon. They have entire teams of people making sure you don't have to deal with the actual complexity. That's the difference. This isn't an actual abstraction in the same way a high level language abstracts the details away from you. This is an abstraction where a team of people take care of you. It is not a true abstraction because you cannot wield this abstraction yourself in the same way you can wield a high level programming language. Make no mistake, the actual complexity of it all has increased by huge orders of magnitude.
  - squabble 5y ago
    I think they mean that it reduces the complexity that a single developer has to deal with.

zwieback 5y ago

A lot of nostalgia for an era that didn't exist: “It has never been more difficult to be a software developer than it is today...” and “Everything was so much simpler before..." SW dev has always been overly complex, long before AWS and the web existed. It's just how we do things in software engineering. Too many choices and lack of physical constraints leads us right to the edge of where things totally fall apart.

bmitc 5y ago
Just because it's always been complex doesn't mean the complexity isn't increasing. Software development is ridiculously complex, increasingly so, and one often doesn't have any tools to deal with it.
- adminscoffee 5y ago
  i think it's done for job security. kinda like antivirus software always had new "threats", but that's probably not true, it just feels sometimes like we are reinventing the wheel to keep tech jobs relevant, at least in certain fields
  - qsort 5y ago
    If this were the case, you'd see at least some people/orgs doing better and spending less than the current "best practice". I don't think it's reasonable to say software developers are just fooling everyone.
pjmlp 5y ago
VB 6/Delphi in a 3-tier application versus HTML/CSS/JS npm spaghetti pretending to be a GUI framework instead of interactive documents.
- deleted 5y ago
  [deleted]

redact207 5y ago

I don't quite agree with much of this article. It paints the industry with a broad brush and not everyone has had the same experience - least of which me. The article starts: > The shift from building applications in a monolithic architecture hosted on a server you could go and touch, to breaking them down into multiple microservices, packaged up into containers, orchestrated with Kubernetes, and hosted in a distributed cloud environment, marks a clear jump in the level of complexity of our software. Add to that expectations of feature-rich, consumer-grade experiences, which are secure and resilient by design, and never has more been asked of developers. Comparing the most complicated way to build and run an app with something that isn't as complicated is just a straw man. Today you can still build a simple, monolithic app and deploy it into a completely managed service. All the points raised address certain problems: - microservices - sliced out of a monolith when complexity or scalability becomes difficult - containerisation - deploy an app with all of its dependencies down right down to the OS - orchestration - ensure reliability when pods/tasks fail, rolling updates, commoditize hardware - distributed cloud environment - remove the need for on-prem, or have the ability to elastically scale without monthly VPS commitments - secure & resilient by design - apps that make money should have good uptime, not leak data None of those are mandatory for all apps - it's pick and choose as and when you get the need. But also what is the alternative provided from "years ago" when things were seemingly simpler? - microservices - ever tried keeping velocity when building XMLSOAP based apps? - containerisation - write a bunch of ansible scripts or maintain disk images that can be sent to server fleets with bash scripts, else more scripts to apt-get/apk install stuff. On windows before package managers? enjoy shipping zips and exes with your code. - orchestration - scripts? - distributed cloud environment - build your own datacentre if you're big enough, lease out some racks if your not and have a systems team maintain them, patch them, RAID stripe disks and swap out dying infrastructure; or sign up to managed VPSs and raise a requisition order each time you want a new server that'll take a week to go through. - secure & resilient by design - a lot to do for a 6-9s uptime; so just lock down your release process with a ton of bureaucracy and approval gates by different teams. not releasing is great to improve uptime SLAs. I don't have a good impression of the "good old days". The complexity was always there, it was just managed by different teams. The real question is should a software developer need to know and work across all of these things, or are people who no longer have "systems management" job move into DevSecOps and augment a software team?

ogazitt 5y ago

Every few years, some version of this article comes out. The version of 30 years ago lamented how programmers don't really understand the hardware anymore because a compiler abstracts the machine. 20 years ago, it was about how virtual machines (like the JVM / CLR) abstract memory management and developers write bloated software. We simply have more layers of abstraction that we depend on, and the interesting problems move up the stack. That trend will continue until the machines take over and write software. Until then, we'll continue to see this article show up, s/kubernetes/next-new-hotness.

erik_seaberg 5y ago
The stack also improves. Early versions of the JVM were very noticeably slower than C++, and the first CLR GC was so poor that my team at Microsoft had to resort to arrays of structs for keeping a service’s state. These days they’re great, and it’s hard to do better without putting a lot more effort into something like Rust.
- adrianN 5y ago
  The JVM is great, but I think it's not too difficult to do better in a language that doesn't force so much pointer-chasing. Cultural differences probably also play a role. Java systems tend to be engineered such that more pointer chasing happens even in places where the language doesn't force it.
  - WalterGR 5y ago
    Does the CLR improve on that much, with its stack-allocated structs?
- jcelerier 5y ago
  > Early versions of the JVM were very noticeably slower than C++, The fastest java software I use feels much slower than the slowest c++ one (talking about UI lag)
  - 5y ago

rglover 5y ago

Microservices and serverless were the "we've jumped the shark" point for me. Both can be a great pattern applied properly (nasty scaling problems), but they became a religion or a replacement for entire stacks. I see so many devs these days who are clearly confused but have no idea what they're doing is overly complex—they just think it's normal. AWS and the like are great, but the move to make everything cloud-based is going to lead to a lot of unmaintainable (or in extreme cases, unsalvageable) messes later. It will be a sad day when devs only understand how to build or deploy something if it's using a heavily-abstracted corpo stack, or even worse, "no code."

adminscoffee 5y ago
no code will increase alongside ai, developers won't have jobs in the future. it'll be interfaces where the system builds it's solution okay after you tell it what you want. basically jit on steroids
- bruce343434 5y ago
  all of the no code creations I have seen posted on this site in the past month are incredibly slow and sluggish. Sometimes even straight up buggy, for instance that font compare picker thing, would sometimes load no font (null) on the left side. As a paradigm, it has a long road to go. If people really think that no code is acceptable, then I am going to cry. It won't be an improvement.
  - rglover 5y ago
    Yes. It's not about improvement, it's about ease of use and reduction of work, or: convenience. That convenience is going to bite an incredible number of people from behind and send a lot of businesses spiraling.
- Ekaros 5y ago
  Will there really be a no code, or is that just moving the level of abstraction one more level higher... While still needing occasional coding to solve those bits that aren't available...

deleted 5y ago

[deleted]

blondin 5y ago

modern software has become complex. but people, especially those involved in this complexity, insist that this is all normal. simplicity (or whatever it will be) is coming for modern software. we are seeing the early stages of that upcoming change: denial & resistance. denial is when we hear engineers say that software is simpler or no more complex than it was. resistance is when engineers try to convince us that "we need that complexity". following the well-known psychological curve, next up is acceptance. and i have started to see it already. kubernetes, for example, continue to deprecate poorly designed or overly complex features. it is hard because of backward compatibility. all that to say that change is coming. maybe we will simplify and start the cycle all over again :)

systemvoltage 5y ago
I am not a fan of denial & resistance arguments. Do bad thing X. Try to convince others that X is good and its failure of acceptance is just people's denial. Next step is the well-known psychological curve that you will accept X eventually. This can be applied indiscriminately to anything one wants to push without having to defend the specifics. We should instead question all assertions - there is no law out there that denial always precedes acceptance.
- CRConrad 5y ago
  > there is no law out there that denial always precedes acceptance. Yeah. "Denial, resistance, acceptance" -- isn't that from the six (or was it seven?) stages of grief? But grief is for something inevitable and irresistible, a fait accompli: When a loved one has died that has already happened and there is nothing you can do about it. Here, though, if enough of us think that our industry is going the wrong way, maybe we can still stop it.

kashyapc 5y ago

For what it's worth, one of the QEMU (the open source emulator and virtualizer) maintainers, Paolo Bonzini, recently gave a talk about this topic. Here's an extended summary of that on LWN: "A QEMU case study in grappling with software complexity" — https://lwn.net/Articles/872321/ https://lwn.net/Articles/872321/ Although the article uses QEMU as an example, the lessons discussed in it are equally applicable to many other projects. (If you're short on time, read the intro, "Sources of complexity", "Ways to fight back", and the short conclusion. The QEMU command-line case study section might be interesting for those who use a lot of QEMU, or to those devs who use QEMU to write higher level tools.)

cjfd 5y ago

The biggest problem seem to be microservices. Basically, an army of cargo culters missed the small print that you should not even be thinking about them unless you work with at least 100 programmers. The talking heads that popularized microservices are also guilty of this. They occasionally mention that microservices are only for large teams but not nearly often enough.

comeonseriously 5y ago
Cargo culting is just what you have to learn to accept and work around, because there will always be developers who fall for it. A decade ago, everyone was writing dependency injection frameworks because that was the hotness and debugging became a nightmare, but, hey!, we completely decouple EVERYTHING! Then there was no sql (mongo etc) and if you were still using sql, your system was shit. Oh and way before that there was GoF and you ended up with UserBillingFacadeFactoryFactory abstractions. None of these technologies are bad. They're just bad in the hands of mediocre developers.
packetlost 5y ago
Yes and no. There are legitimate reasons to build microservices with small teams (such as requiring differing runtime environments due to hardware/architectural constraints). They should never be the default at a team below that scale though.

davidgl 5y ago

Is it more complicated though ? I've been developing for over 30 years, and I don't feel like its got any worse, I think it's the nature of the medium. These days I fight with npm or nuget dependencies problems, 20 years ago I was trying to write Delphi code to do simple REST requests, the complexity moves around but is still there. Out of the Tar Pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf) was written 15 years ago, and it's all about curbing complexity

ratww 5y ago
I think this is an example of simple vs easy. Writing request code 20 years ago was harder, because you needed to read docs and write a lot by hand. But it was simpler because it didn't add any additional hidden complexity from libraries or whatnot. At least in most languages it got better: we only need something simple and easy, like Javascript's fetch. Does complexity we don't have to maintain count? About Out of the Tar Pit, complexity manifests itself in various shapes. The essay itself lists some that have gotten better: state is one, with people using more functional code. The "Power corrupts" part has also gotten better, IMO, with languages becoming simpler (Go) or more restrictive (Rust). I think the "Code Volume" part is still a big one. One thing I'd like to see curbed to avoid code volume are the proliferation of code for handling edge cases. Having libraries and code able to handle tens of edge cases is both a blessing a curse. However most developers and business people can only see the blessing part.

lbriner 5y ago

As other posters have said, I think there is somewhat too much nostalgia about how "easy" it was in the old days. I was reading about software dev in the 70s and it took like 4 years to write what we could do in a day today. Also, I think that relatively, we are still in the early days of software where lots of people are clamouring to invent the silver bullet to make it all work better, but after a bit longer, we will have more experience of what works and doesn't work and we will consign certain things to history. Think about what medicine was like in the first few centuries? Leeches, trepanning, toxic "medicines", they presumably had the same issues until they established a much better foundation of what works and what doesn't which has made medicine move on. We will keep some of our cool things but others (XML anyone!) will just die off as we realise they are a solution to a problem that most of us don't have.

v-erne 5y ago
>>Think about what medicine was like in the first few centuries ... I spend thinking about medicine quite a lot (health problems in my family run deep) and this does not give me great deal of hope for our industry at all. There is so much quackery everywhere - most of diagnostics is based on some magic guess work. If symptoms are non-specific than You are out of luck - You have to spend years visiting random doctors hoping that one of them will have fresh enough memory of some book that he read years ago to guess right. And even when the diagnosis is easy the therapy can be not that far away from leeches. Few years ago I had wart on my hand and dermatologist offered me three options: - freeze it to death - burn it to death with acid - cut it out If You look at this options closely they require some high tech to be possible but from medical point of view they are as primitive as leeches. I honestly belive that in 100 years time when nanorobots will be available to everyone the doctor will prescrabie therapy that is basically nanorobots attacking the wart with little flamethrowers.
fn1 5y ago
> We are still in the early days of software where lots of people are clamouring to invent the silver bullet to make it all work better [...] I have a feeling that (especially in devOps/kubernetes) often there is a wish to have a magic button/command that does everything. Like in a rube-goldberg machine. What is forgotten then is the cognitive overhead of templating YAML inside templating YAML which is needed for the magic-button to be configured. And if the buttons behaviour needs to be changed very few weeks, not much is gained.

sswastioyono18 5y ago

some mention it about microservices but no one mention about the emerge of cloud? when we are still using monolith and using old deploy to one server, that would be easy. Then there is scaling issue and that's where the cloud came because really, when you have to manage a lot of things at once, it's better to let someone manage it for you and you can focus on your business logic. Obviously, the practice wasn't perfect. A lot of "micronization" comes up instead domain driven problem but that's the problem on mostly people not the tech

aryehof 5y ago

Perhaps this suits todays technology focused developer. There has been no progress on how to effectively determine and model complex business domain requirements into code, so just focus on the technological requirements and plumbing.