4 ms·

> It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things. > I think in 2022 we're

> It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things. > I think in 2022 we're back in a similar situation again. Virtualisation, containers, cloud services and whatnot have regressed development reliability. Realistically you probably want to find the middle ground between these: not copying PHP code through SFTP or even working on it directly, but also not running an enterprise Kubernetes cluster for your simple CRUD app. Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). > I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it." That sort of happened with Roblox, they had an outage that went on for 3 days due to Nomad rolling over and dying in their infra and nobody really knew what the problem was or how to fix it for quite a while: https://roblox.fandom.com/wiki/2021_Roblox_outage https://roblox.fandom.com/wiki/2021_Roblox_outage From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.

Sharlin 5y ago
> Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). This is essentially why I’m now in the midst of a career change. I realized I want no part in what is more of a cargo cult than an engineering discipline.
- _notathrowaway 5y ago
  I am curious, if not too personal, would you mind sharing what do you plan on working on?
  - Sharlin 5y ago
    I recently enrolled in a newly launched BSc program at my local university titled Sustainable Urban Development, blending administrative and social sciences with engineering in an attempt to tackle current and future issues related to urbanization, urban planning and sustainable development. https://www.tuni.fi/en/study-with-us/technology-sustainable-urban-development https://www.tuni.fi/en/study-with-us/technology-sustainable-...
    - _notathrowaway 5y ago
      Good luck!
      - Sharlin 5y ago
        Thanks :)
sbarre 5y ago
> From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction. I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. A team is left behind to operate that infrastructure, but they may not necessarily understand it to the depth required to react properly when it fails or when it needs updating, etc.. There's a 20+ year old book called The Ingenuity Gap that addresses this exact challenge with complex systems that we aren't equipped to do much more than "keep the lights on".. It definitely feels like we are running up against similar situations in the software world, particularly with the huge amount of job churn (and corporate brain drain) that the pandemic produced.
- ghaff 5y ago
  You see this at all different levels of complexity. I'm on the board of a very small non-profit and periodically I see someone come in with grand plans for updating the website and other infrastructure. And you pretty much know that, given free rein, they'll come up with something that will be way beyond what the person coming in after them will be in a position to maintain.
  - syntheticnature 5y ago
    As a volunteer at a small non-profit attempting to keep a similar-sounding setup working long enough to replace it, this is the nightmare I am attempting to unwind. The problem when moving to off-the-shelf software for things is everyone wanting their particular favorite features. Said features were all custom-coded by someone without formal development training, of course.
- notyourday 5y ago
  > I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. Most companies do not need complex infrastructure either. It is just cargo culting ( or Google culting )
- hinkley 5y ago

Sharlin 5y ago

> Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes). This is essentially why I’m now in the midst of a career change. I realized I want no part in what is more of a cargo cult than an engineering discipline.

_notathrowaway 5y ago
I am curious, if not too personal, would you mind sharing what do you plan on working on?
- Sharlin 5y ago
  I recently enrolled in a newly launched BSc program at my local university titled Sustainable Urban Development, blending administrative and social sciences with engineering in an attempt to tackle current and future issues related to urbanization, urban planning and sustainable development. https://www.tuni.fi/en/study-with-us/technology-sustainable-urban-development https://www.tuni.fi/en/study-with-us/technology-sustainable-...
  - _notathrowaway 5y ago
    Good luck!
    - Sharlin 5y ago
      Thanks :)

sbarre 5y ago

> From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction. I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. A team is left behind to operate that infrastructure, but they may not necessarily understand it to the depth required to react properly when it fails or when it needs updating, etc.. There's a 20+ year old book called The Ingenuity Gap that addresses this exact challenge with complex systems that we aren't equipped to do much more than "keep the lights on".. It definitely feels like we are running up against similar situations in the software world, particularly with the huge amount of job churn (and corporate brain drain) that the pandemic produced.

ghaff 5y ago
You see this at all different levels of complexity. I'm on the board of a very small non-profit and periodically I see someone come in with grand plans for updating the website and other infrastructure. And you pretty much know that, given free rein, they'll come up with something that will be way beyond what the person coming in after them will be in a position to maintain.
- syntheticnature 5y ago
  As a volunteer at a small non-profit attempting to keep a similar-sounding setup working long enough to replace it, this is the nightmare I am attempting to unwind. The problem when moving to off-the-shelf software for things is everyone wanting their particular favorite features. Said features were all custom-coded by someone without formal development training, of course.
notyourday 5y ago
> I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. Most companies do not need complex infrastructure either. It is just cargo culting ( or Google culting )
hinkley 5y ago