6 ms·

This project was killed by invisible complexity. I felt a little sad reading about it. There are lots of comments talking about the sustainability of projects w

This project was killed by invisible complexity. I felt a little sad reading about it. There are lots of comments talking about the sustainability of projects with respect to money and burnout, the need to pay for services etc. I get that and agree. But I also get the impression that wouldn't have helped here. When we reach a situation where some unknown person pushes an upstream update, which causes a cascade of problems that even someone with deep technical knowledge and daily connection to the project cannot unpick, that's a breakdown in modularity/dependency. It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things. I think in 2022 we're back in a similar situation again. Virtualisation, containers, cloud services and whatnot have regressed development reliability. I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."

ehnto 5y ago
Excellent comment and I can't speak directly for the author, but that's what I read too. Incidental Complexity is a term related to this. Incidental Complexity is any complexity that is not part of solving the core domain problem and has been introduced by the tooling or approach you chose instead. The most common example I see is using enterprise tooling for small teams and projects, and then those projects getting snowed under by the weight of the hidden and incidental complexity they had built without noticing.
- Sharlin 5y ago
  Essential vs incidental complexity is one of those concepts everybody who wants to call themselves a software engineer should be intimately familiar with. As they say, anyone can design a bridge, but it takes an engineer to just barely design a bridge.
- ignoramous 5y ago
  Incidental Complexity pushes me towards serverless offerings. Of course, serverless brings with it its own test/deploy quagmire, but I'd argue for personal projects, it doesn't matter as much.
lbriner 5y ago
True but there is also a difference in priority with different package maintainers which isn't always helpful. Why were old features removed from the upstream packages? Probably because upstream either wanted to ditch old/insecure protocols or the usage was too low to warrant the investment in keeping it alive. That is completely reasonable for them. Then look at something like SSLPing, it wants to support old protocols because that is part of the feature set for older servers which have no realistic need to update. It's something I find a bit troubling about the very vocal "security" crowd for whom only the latest and greatest ciphers/protocols/short-lived ssl certs/stronger DH key pairs are acceptable and everything else is garbage. Sadly, it seems that realistically, only paying/charging will help this because then it is reasonable for someone to spend 40 hours of their week keeping up with the changes/building their own libraries/etc. to keep their project running.
nikanj 5y ago
"I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."" That day is today: https://confluence.status.atlassian.com/incidents/hf1xxft08nj5 https://confluence.status.atlassian.com/incidents/hf1xxft08n...
- vincnetas 5y ago
  6 days and counting
- djfobbz 5y ago
  Insanity at its best! Atlassian annual revenue for 2021 was $2.089B, a 29.42% increase from 2020. Atlassian annual revenue for 2020 was $1.614B, a 33.39% increase from 2019. Atlassian annual revenue for 2019 was $1.21B, a 37.36% increase from 2018.
- dahart 5y ago
  Nah, Atlassian has not given up, and isn’t going to. @nonrandomstring wasn’t referring to incidents that take more than a minute to resolve, they were predicting an incident of such magnitude that a large company shuts down over it. This is definitely not that.
  - paulclinger 5y ago
    I read it differently: the company will not shut down, but abandon attempts to resolve the issue (whether for a product or a set of customers) due to its complexity. Using major in GP's description was to highlight that the company has necessary resources and still can't figure out what's going on.
    - dahart 5y ago
      Fair enough, there is a valid more optimistic interpretation than mine. Still, this is not even the case you’re suggesting. Atlassian isn’t abandoning any features or customers here, this case isn’t indicative of anything other than a run of the mill outage that’s taking longer than a day to fix.
KronisLV 5y ago
> It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things. > I think in 2022 we're back in a similar situation again. Virtualisation, containers, cloud services and whatnot have regressed development reliability. Realistically you probably want to find the middle ground between these: not copying PHP code through SFTP or even working on it directly, but also not running an enterprise Kubernetes cluster for your simple CRUD app. Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). > I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it." That sort of happened with Roblox, they had an outage that went on for 3 days due to Nomad rolling over and dying in their infra and nobody really knew what the problem was or how to fix it for quite a while: https://roblox.fandom.com/wiki/2021_Roblox_outage https://roblox.fandom.com/wiki/2021_Roblox_outage From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.
- Sharlin 5y ago
  > Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). This is essentially why I’m now in the midst of a career change. I realized I want no part in what is more of a cargo cult than an engineering discipline.
  - _notathrowaway 5y ago
    I am curious, if not too personal, would you mind sharing what do you plan on working on?
    - Sharlin 5y ago
dahart 5y ago
> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it." On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity, and they will have had hundreds of service outages. This is in stark contrast to the single-user free service we’re commenting on, where the owner was already treading water and perhaps just didn’t know it, and where the tradeoffs never had financial upside. On the other hand, you’re not just right about your prediction, but it has already happened in the broader sense, just maybe not over a single incident. Complexity has already killed many software companies, when the software gets too big to manage and can’t be easily refactored, runs on old languages and tools that stop getting updates, relies on too many services that slowly go out of both fashion and support contracts. The list of companies that got too invested in a single codebase and stuck themselves with the inability to adapt quickly enough due to the weight and complexity of what they built is pretty long and growing.
- nonrandomstring 5y ago
  > "On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity." Absolutely the companies themselves can be very robust. Some friends and colleagues who work for <big defence> tell me they don't fear for their jobs if a project is crashing. Internally such firms are structured to run dozens of projects, all well compartmentalised, such that if one fails they'll just get moved to another. The company itself is survivable via diversity (plus there's .gov money close at hand to bail out in an emergency). What concerns me about civil resilience is where we've got unexamined mono-cultures like Google Docs or GitHub that become de-facto pillars for countless other businesses. When Trump ordered Adobe to cut-off Venezuela that should have been a resilience wake-up call for everybody (and it prefigures the present Russian situation of course). There could be many reasons (more likely an unfortunate confluence of unrelated upstream reasons) that topple a vital part of public infrastructure. That's why big-tech monopoly, which erodes hybrid resilience, bothers the hell out of me.
commandlinefan 5y ago
> we reach a situation where some unknown person We reach an unnecessary situation. The vast majority of these dependencies don't even add any real value - not enough to justify the danger of having your app just stop working inexplicably.
hedora 5y ago
Yeah; I was going to write that this shutdown notice is a great microcosm for what's wrong with the field. You focus on containers, but this project had dependencies fail at each level. The author also cites init breaking, the crypto library losing support for old protocols, node breaking compatibility, and I think some frontend junk I haven't heard of also fell out of support. The project sounds like it could have been a well-scoped command line option to SSL or something, but, because it was built with current best practices, keeping the dependencies up to date and maintaining it would probably take multiple full time engineers / swe's at a FAANG. On top of that, there's no sustainable business models for good ?aaS tools (small, or large), just for platforms and monopolistic stuff.
- furyofantares 5y ago
  > because it was built with current best practices, keeping the dependencies up to date and maintaining it would probably take multiple full time engineers / swe's at a FAANG. Surely significantly less than one full time? But like 10% of an extra full time job that costs you money isn't a very good proposition
  - hedora 5y ago
    Each dependency gets to choose when they break your service (they can have a zero day exploit that's only fixed in a compatibility breaking release, for example). You need two people if you want one to be able to go on vacation while you maintain a 24 hour SLA for security patches. You need a third because this sort of thing generates employee burn out once it is feature complete. You could try having it be 10% of one employee, but that employee would need to be a full stack expert from low level Linux stuff to JavaScript GUI junk. Good luck finding two people like that, then telling them they own ten feature complete services like this. The people that fit such a bill and have the personaility type to be happy in such a job are expensive and rare!
    - crabmusket 5y ago
      > 24 hour SLA for security patches I think that's a significant step up from your earlier guess at "maintaining" the software. There's a big spectrum between "project has been shut down" and "round the clock SLA".
ianbutler 5y ago
https://status.atlassian.com/ https://status.atlassian.com/ Atlassian has had a major incident ongoing for about 6 days now, so we may be at that point sooner than you were hoping for.
rsync 5y ago
"When we reach a situation where some unknown person pushes an upstream update, which causes a cascade of problems that even someone with deep technical knowledge and daily connection to the project cannot unpick, that's a breakdown in modularity/dependency." Agreed. I believe that the "UNIX philosophy" is a good strategy for avoiding these scenarios and I believe it is underappreciated - even by architects and designers that should know better. Forgive me for injecting this word but this is why I recoil so violently from systemd and all of its accoutrements. I can sense how this is going to fail even if I can't clearly elucidate it just yet ...
systemvoltage 5y ago
Why do we need Docker? Just so that it’s easy to ship to server? How about rsync + taking the risk of dependencies conflicting? Sorry not expert at all the infra stuff. But I hope we can question things.
ilaksh 5y ago
Nonsense. It was a side project that cost a lot of money. He had to take care of his actual job, family, hobbies, income producing side projects, etc. He had no single person on the planet to help him maintain it or update it. So the dependencies went out of date over the years. Projects with resources can be maintained and updated. This one has no resources and there is no reason to update it because it's a drag on that person's life.

ehnto 5y ago

Excellent comment and I can't speak directly for the author, but that's what I read too. Incidental Complexity is a term related to this. Incidental Complexity is any complexity that is not part of solving the core domain problem and has been introduced by the tooling or approach you chose instead. The most common example I see is using enterprise tooling for small teams and projects, and then those projects getting snowed under by the weight of the hidden and incidental complexity they had built without noticing.

Sharlin 5y ago
Essential vs incidental complexity is one of those concepts everybody who wants to call themselves a software engineer should be intimately familiar with. As they say, anyone can design a bridge, but it takes an engineer to just barely design a bridge.
ignoramous 5y ago
Incidental Complexity pushes me towards serverless offerings. Of course, serverless brings with it its own test/deploy quagmire, but I'd argue for personal projects, it doesn't matter as much.

lbriner 5y ago

True but there is also a difference in priority with different package maintainers which isn't always helpful. Why were old features removed from the upstream packages? Probably because upstream either wanted to ditch old/insecure protocols or the usage was too low to warrant the investment in keeping it alive. That is completely reasonable for them. Then look at something like SSLPing, it wants to support old protocols because that is part of the feature set for older servers which have no realistic need to update. It's something I find a bit troubling about the very vocal "security" crowd for whom only the latest and greatest ciphers/protocols/short-lived ssl certs/stronger DH key pairs are acceptable and everything else is garbage. Sadly, it seems that realistically, only paying/charging will help this because then it is reasonable for someone to spend 40 hours of their week keeping up with the changes/building their own libraries/etc. to keep their project running.

nikanj 5y ago

"I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."" That day is today: https://confluence.status.atlassian.com/incidents/hf1xxft08nj5 https://confluence.status.atlassian.com/incidents/hf1xxft08n...

vincnetas 5y ago
6 days and counting
djfobbz 5y ago
Insanity at its best! Atlassian annual revenue for 2021 was $2.089B, a 29.42% increase from 2020. Atlassian annual revenue for 2020 was $1.614B, a 33.39% increase from 2019. Atlassian annual revenue for 2019 was $1.21B, a 37.36% increase from 2018.
dahart 5y ago
Nah, Atlassian has not given up, and isn’t going to. @nonrandomstring wasn’t referring to incidents that take more than a minute to resolve, they were predicting an incident of such magnitude that a large company shuts down over it. This is definitely not that.
- paulclinger 5y ago
  I read it differently: the company will not shut down, but abandon attempts to resolve the issue (whether for a product or a set of customers) due to its complexity. Using major in GP's description was to highlight that the company has necessary resources and still can't figure out what's going on.
  - dahart 5y ago
    Fair enough, there is a valid more optimistic interpretation than mine. Still, this is not even the case you’re suggesting. Atlassian isn’t abandoning any features or customers here, this case isn’t indicative of anything other than a run of the mill outage that’s taking longer than a day to fix.

KronisLV 5y ago

> It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things. > I think in 2022 we're back in a similar situation again. Virtualisation, containers, cloud services and whatnot have regressed development reliability. Realistically you probably want to find the middle ground between these: not copying PHP code through SFTP or even working on it directly, but also not running an enterprise Kubernetes cluster for your simple CRUD app. Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). > I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it." That sort of happened with Roblox, they had an outage that went on for 3 days due to Nomad rolling over and dying in their infra and nobody really knew what the problem was or how to fix it for quite a while: https://roblox.fandom.com/wiki/2021_Roblox_outage https://roblox.fandom.com/wiki/2021_Roblox_outage From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.

Sharlin 5y ago
> Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). This is essentially why I’m now in the midst of a career change. I realized I want no part in what is more of a cargo cult than an engineering discipline.
- _notathrowaway 5y ago
  I am curious, if not too personal, would you mind sharing what do you plan on working on?
  - Sharlin 5y ago

dahart 5y ago

> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it." On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity, and they will have had hundreds of service outages. This is in stark contrast to the single-user free service we’re commenting on, where the owner was already treading water and perhaps just didn’t know it, and where the tradeoffs never had financial upside. On the other hand, you’re not just right about your prediction, but it has already happened in the broader sense, just maybe not over a single incident. Complexity has already killed many software companies, when the software gets too big to manage and can’t be easily refactored, runs on old languages and tools that stop getting updates, relies on too many services that slowly go out of both fashion and support contracts. The list of companies that got too invested in a single codebase and stuck themselves with the inability to adapt quickly enough due to the weight and complexity of what they built is pretty long and growing.

nonrandomstring 5y ago
> "On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity." Absolutely the companies themselves can be very robust. Some friends and colleagues who work for <big defence> tell me they don't fear for their jobs if a project is crashing. Internally such firms are structured to run dozens of projects, all well compartmentalised, such that if one fails they'll just get moved to another. The company itself is survivable via diversity (plus there's .gov money close at hand to bail out in an emergency). What concerns me about civil resilience is where we've got unexamined mono-cultures like Google Docs or GitHub that become de-facto pillars for countless other businesses. When Trump ordered Adobe to cut-off Venezuela that should have been a resilience wake-up call for everybody (and it prefigures the present Russian situation of course). There could be many reasons (more likely an unfortunate confluence of unrelated upstream reasons) that topple a vital part of public infrastructure. That's why big-tech monopoly, which erodes hybrid resilience, bothers the hell out of me.

commandlinefan 5y ago

> we reach a situation where some unknown person We reach an unnecessary situation. The vast majority of these dependencies don't even add any real value - not enough to justify the danger of having your app just stop working inexplicably.

hedora 5y ago

Yeah; I was going to write that this shutdown notice is a great microcosm for what's wrong with the field. You focus on containers, but this project had dependencies fail at each level. The author also cites init breaking, the crypto library losing support for old protocols, node breaking compatibility, and I think some frontend junk I haven't heard of also fell out of support. The project sounds like it could have been a well-scoped command line option to SSL or something, but, because it was built with current best practices, keeping the dependencies up to date and maintaining it would probably take multiple full time engineers / swe's at a FAANG. On top of that, there's no sustainable business models for good ?aaS tools (small, or large), just for platforms and monopolistic stuff.

furyofantares 5y ago
> because it was built with current best practices, keeping the dependencies up to date and maintaining it would probably take multiple full time engineers / swe's at a FAANG. Surely significantly less than one full time? But like 10% of an extra full time job that costs you money isn't a very good proposition
- hedora 5y ago
  Each dependency gets to choose when they break your service (they can have a zero day exploit that's only fixed in a compatibility breaking release, for example). You need two people if you want one to be able to go on vacation while you maintain a 24 hour SLA for security patches. You need a third because this sort of thing generates employee burn out once it is feature complete. You could try having it be 10% of one employee, but that employee would need to be a full stack expert from low level Linux stuff to JavaScript GUI junk. Good luck finding two people like that, then telling them they own ten feature complete services like this. The people that fit such a bill and have the personaility type to be happy in such a job are expensive and rare!
  - crabmusket 5y ago
    > 24 hour SLA for security patches I think that's a significant step up from your earlier guess at "maintaining" the software. There's a big spectrum between "project has been shut down" and "round the clock SLA".

ianbutler 5y ago

https://status.atlassian.com/ https://status.atlassian.com/ Atlassian has had a major incident ongoing for about 6 days now, so we may be at that point sooner than you were hoping for.

rsync 5y ago

"When we reach a situation where some unknown person pushes an upstream update, which causes a cascade of problems that even someone with deep technical knowledge and daily connection to the project cannot unpick, that's a breakdown in modularity/dependency." Agreed. I believe that the "UNIX philosophy" is a good strategy for avoiding these scenarios and I believe it is underappreciated - even by architects and designers that should know better. Forgive me for injecting this word but this is why I recoil so violently from systemd and all of its accoutrements. I can sense how this is going to fail even if I can't clearly elucidate it just yet ...

systemvoltage 5y ago

Why do we need Docker? Just so that it’s easy to ship to server? How about rsync + taking the risk of dependencies conflicting? Sorry not expert at all the infra stuff. But I hope we can question things.

ilaksh 5y ago

Nonsense. It was a side project that cost a lot of money. He had to take care of his actual job, family, hobbies, income producing side projects, etc. He had no single person on the planet to help him maintain it or update it. So the dependencies went out of date over the years. Projects with resources can be maintained and updated. This one has no resources and there is no reason to update it because it's a drag on that person's life.