10 ms·

In the late 90s I travelled the world with a couple of CD-ROMs installing NT 4.0 on individual physical servers. I understood the entire stack. I was the only e

In the late 90s I travelled the world with a couple of CD-ROMs installing NT 4.0 on individual physical servers. I understood the entire stack. I was the only engineer and I was often very remote. In the mid 2000s we installed Server 2003/8 on a whole bunch of physical servers in a data centre. This was for one of Australia’s ‘big four’ banks. Me and a team of less than ten, most of whom I still know, managed the entire thing. We were the ‘3rd level infrastructure’ team. Most of us understood most of the stack end-to-end, although there were some specialities. In 2023 I work for an IT integrator. These days we use Azure, because Microsoft has fooled us in to thinking that it’s cheaper. My current project is, essentially, a website migration. Not even a big or complex website. The project has been running for 6 months. I only joined recently so I can’t be sure but we’ve probably burned AU$2m. The schedule I’m managing has us doing the cutover in June. That is optimistic. The architects are currently trying to work out how Azure [details] connect to Azure [details] while still [security] and being able to [complex integration]. Every day some new issue appears that the architects have to figure out how to work around. No single person has a goddamned clue how the end-to-end thing fits together. Not one, not a clue. That scares me. ‘Architectural complexity’ has crept down to the level of infrastructure. It’s painful to see. I hope it stops soon but I am not hopeful.

lifeisstillgood 4y ago
This is the essence - that a system needs to fit inside one persons head. All of it. There maybe a few people who understand all of a Boeing airplanes systems - there certainly were when they first flew 747s. Maybe a couple now. but once it stops fitting inside one persons head then the thing is literally only possible to design by committee - it cannot fit together and perhaps should stop being called a single system
- bob1029 4y ago
  > it cannot fit together and perhaps should stop being called a single system The magic is the enterprise message bus. It is meme crap until you actually need one. When it works, it really works. This is the only thing I've ever seen realistically tie together certain industrial environments.
- cortesoft 4y ago
  There is no one who understands FULLY how a single computer works (at least not to the specific levels of detail on how to make every part from scratch... how to mine the heavy metals, how to chemically process them, etc) That bas been impossible for hundreds of years. The important part is where you draw up the system boundaries
  - concordDance 4y ago
    > That bas been impossible for hundreds of years. You mean tens of years?
    - cortesoft 4y ago
      For computers specifically, sure, but I was talking about almost any complex machine in the last few hundred years.
      - MichaelZuo 4y ago
        I'm fairly certain the chief engineer on the System 360 understood everything down to the transistors. Anything past that though... maybe QNX down to the bytecode? It's at least elegant enough to suggest so.
        cortesoft 4y ago
        You think they knew how to mine the ore and process it into usable material to manufacture those transistors? You might argue that that isn't required to fully understand the system, but that argument is no different than saying you don't need to understand the hardware at a detailed level to fully understand your system.
        MichaelZuo 4y ago
        I lived across the park from the best mining school in Canada so yes I believe that's totally possible. A few months of intensive study, or a year or 2 of more casual study is all that's needed with the right guidance.
throwawaaarrgh 4y ago
Automation moves complexity from one place to another. In this case the complexity moved into silos. Before, all the pieces of the architecture were in a couple hundred *.cpp files in one directory hierarchy. Now it's millions of files across thousands of directories each turned into a service run by a dedicated team, and none of them can see each other. You can become an expert in all those pieces. But it requires actually using them all together, to discover the parts that don't fit and how to work around them. This is why the modern ideal of completely independent APIs is a terrible design. There is literally no way to know if anything works with anything else until you try to run it in production. Monoliths are terrible at scaling, but easy to understand. SOA apps are great at scaling, but impossible to understand. Good luck on your migration. Changing the wheels on a moving tractor trailer always sucks.
- lifeisstillgood 4y ago
  Isn't the point that there is a line somewhere that one app stops being one single app and becomes "eco system" - and you stop having deterministic understanding and start having "town planning" and "social expectations". I mean my use of bad metaphors kind of underlines the point we don't really understand his problem - but the large organisation that builds these is itself an example of such impossible to understand interactions - maybe we will learn from Azure etc and take those learnings into taking our own orgs.
  - throwawaaarrgh 4y ago
    Well anything designed as SOA is just Conway's Law enshrined in architecture. Most older software was too. The org is the problem. The super bizarre outlier is the Open Source community, which doesn't really follow any organizational plan (except the corporate projects and foundations and committees that pretend they're open source). For whatever reason, Open Source succeeds despite no hierarchy, no clearly defined architecture, no particular strategy. But then again, maybe that's why it works so well. An organization has a fixed structure, rules, priorities. It can fail, lose its intellectual property, its funding, be competed against. Open Source has no such limits. It's the wild west. Do whatever you want, whenever you want, and whatever works, survives. Lke natural evolution. An organic ecosystem, rather than a plastic one.
vsareto 4y ago
I wouldn't be surprised if most people don't learn it in-depth because: A) you can google or ChatGPT things more easily these days B) there's a truck load of services and ways to connect and do things. It's hard to learn all of these or pick a best option for everyone C) SSO + identity models + API permissions generally made everything more complex D) Many things in Azure are changing often, so you don't want to get too invested in a particular "version" So things got more in-depth while expanding your available options (which are usually just as complex)
- jen729w 4y ago
  I think people don’t learn it [all] in-depth because to do so is essentially impossible. Back in the bank days all of understood Windows, multi-tier AD, DNS, DHCP, SMS* (now MECM via SCCM), that distributed file system whose name I forget, file & print, Exchange, and all the other things that had a plugin to mmc.exe — we could do ‘em all. Oh and a handful of networking because that was also much simpler. Oh and all of the client configuration. You could give me a server and a day and I could stand you up a basic infrastructure. I could know it all. (*Though the maniacal way that SMS 2.0 did its magic with text-based log files shuttled in from the endpoint always drove me bonkers.)
  - Jupe 4y ago
    And, by the time you "know it all", the playing field has changed. RESTful APIs? Nope, we are going full-gRPC. No hand-rolled OAuth - we are using AuthSolution1. Forget that, AuthSolution1 has a security vulnerability, we are going with AuthSolution2. Wait, AuthSolution2 won't work with cloud providers database offering... let's try AuthSolution3. Let's use IFTTT. No, how about Zapier. But, we need gRPC and now GraphQL. The databases are over-provisioned, lets look at serverless as a critical path item. Executive VP #2 thinks we should use a NoSQL solution. BTW: Cloud provider X is dropping support for database version 1.1, we need to migrate to 2.0... or skip ahead to the just-released 3.0. And, then the CEO comes and asks: "Can you spend some time trying to get ChatGPT to build this whole thing for us?"
    - 4y ago
jiggawatts 4y ago
We might be working on the same project... or the "same" project, if you know what I mean. The level of abstraction that is commonplace these days is insane. At $dayjob, there is a reverse proxy in front of a API gateway that is itself a load-balanced service. There are load-balancers behind it, pointing at a reverse proxy on a service fabric. Within that, some "architect" decided to add additional "mid tier" servers because that was the cool thing to do in the 1990s. Behind all of this is an Azure App Service, which is in turn a load-balanced server that includes a reverse proxy. They want to move this to Kubernetes, so they can now have proxies seven layers deep, nested hypervisors running containers, and services ping-ponging across six zones (data centre locations) in two different clouds. That connectivity will go through a software virtual router platform, so it's not even direct point-to-point connectivity. I pity the poor fool who will have to diagnose an operational issue in this madness. No mere mortal will be able to make it go fast.
- userbinator 4y ago
  Someone is paying for all of that, and someone is getting paid. Look no further beyond those getting paid to find the reason such monstrosities exist.
  - nosianu 4y ago
    No I don't think so. That explanation only works for simple systems, if at all. But all the many players and their very diverse needs and ideas are very complex. I think i's more likely that it's something described in studies about human problem solving, which leads us to preferring overly complex solutions because we prefer adding something. https://www.nature.com/articles/d41586-021-00592-0 https://www.nature.com/articles/d41586-021-00592-0 (you may need Sci-Hub - paywalled) > Adding is favoured over subtracting in problem solving > A series of problem-solving experiments reveal that people are more likely to consider solutions that add features than solutions that remove them, even when removing features is more efficient. https://www.nature.com/articles/s41586-021-03380-y https://www.nature.com/articles/s41586-021-03380-y > People systematically overlook subtractive changes https://www.scientificamerican.com/article/our-brain-typically-overlooks-this-brilliant-problem-solving-strategy/ https://www.scientificamerican.com/article/our-brain-typical... https://www.washingtonpost.com/business/2021/04/16/bias-problem-solving-nature/ https://www.washingtonpost.com/business/2021/04/16/bias-prob... I think we also have forces in play that add to this bias. Or they are the result of it, for some of them. For example, when there is an easy and cheap or even free solution there is no incentive for marketing and selling it. It's hard to tell a customer that they don't actually need you, especially if it isn't a one-off, or that they should go somewhere cheaper and more simple. It is much easier to get paid for adding than for doing nothing or for subtracting, both as an employee and as a business. It certainly looks that way when I look around. We have frequent complaints about a lack of skilled labor - but that's also because everything keeps getting more and more complex. For example, once it was enough to once every half year update a piece of paper on each bus or tram or train station with the schedule. Now it's some highly complex networked computer solution and display to display when the next bus is going to arrive. Sure there is added value - but there also is a gigantic amount of new complexity for every small "convenience". When it comes to something like waiting for the next tram, specially in an urban environment with 10 minute schedule, the benefit of knowing if it's 2 or 7 minutes that you'll have to wait is negligible compared to the effort of providing and maintaining that service. (we have this situation in my town). Of course there is added value for much of it, maybe all. But I see the complexity and size of systems I knew in a much more simple form when I grew up exploding. The problem is, continuously educating new people to know ever more and to be able to handle more and more sophisticated systems has a continuously increasing cost to society too, since we don't live any longer than before. Never mind everything we do, from planning to maintenance, is ever more complex too. In addition, the new system may not be as robust as the more simple ones of the past. It all works as long as it does. If we see another chip crisis for example, and I just read a pretty dark forecast for China/Taiwan, all those solutions that require working world-wide supply chains, here, chips from Taiwan, are a problem.
userbinator 4y ago
These days we use Azure, because Microsoft has fooled us in to thinking that it’s cheaper. All the cloud providers do that. You could say that peoples' judgment has been clouded... More seriously, this is how they operate; complexity is a good thing to them, because it means more lock-in and ways to charge you in non-obvious ways, and the whole "consulting" aspect of the industry feeds off that.

lifeisstillgood 4y ago

This is the essence - that a system needs to fit inside one persons head. All of it. There maybe a few people who understand all of a Boeing airplanes systems - there certainly were when they first flew 747s. Maybe a couple now. but once it stops fitting inside one persons head then the thing is literally only possible to design by committee - it cannot fit together and perhaps should stop being called a single system

bob1029 4y ago
> it cannot fit together and perhaps should stop being called a single system The magic is the enterprise message bus. It is meme crap until you actually need one. When it works, it really works. This is the only thing I've ever seen realistically tie together certain industrial environments.
cortesoft 4y ago
There is no one who understands FULLY how a single computer works (at least not to the specific levels of detail on how to make every part from scratch... how to mine the heavy metals, how to chemically process them, etc) That bas been impossible for hundreds of years. The important part is where you draw up the system boundaries
- concordDance 4y ago
  > That bas been impossible for hundreds of years. You mean tens of years?
  - cortesoft 4y ago
    For computers specifically, sure, but I was talking about almost any complex machine in the last few hundred years.
    - MichaelZuo 4y ago
      I'm fairly certain the chief engineer on the System 360 understood everything down to the transistors. Anything past that though... maybe QNX down to the bytecode? It's at least elegant enough to suggest so.
      - cortesoft 4y ago
        You think they knew how to mine the ore and process it into usable material to manufacture those transistors? You might argue that that isn't required to fully understand the system, but that argument is no different than saying you don't need to understand the hardware at a detailed level to fully understand your system.
        MichaelZuo 4y ago
        I lived across the park from the best mining school in Canada so yes I believe that's totally possible. A few months of intensive study, or a year or 2 of more casual study is all that's needed with the right guidance.

throwawaaarrgh 4y ago

Automation moves complexity from one place to another. In this case the complexity moved into silos. Before, all the pieces of the architecture were in a couple hundred *.cpp files in one directory hierarchy. Now it's millions of files across thousands of directories each turned into a service run by a dedicated team, and none of them can see each other. You can become an expert in all those pieces. But it requires actually using them all together, to discover the parts that don't fit and how to work around them. This is why the modern ideal of completely independent APIs is a terrible design. There is literally no way to know if anything works with anything else until you try to run it in production. Monoliths are terrible at scaling, but easy to understand. SOA apps are great at scaling, but impossible to understand. Good luck on your migration. Changing the wheels on a moving tractor trailer always sucks.

lifeisstillgood 4y ago
Isn't the point that there is a line somewhere that one app stops being one single app and becomes "eco system" - and you stop having deterministic understanding and start having "town planning" and "social expectations". I mean my use of bad metaphors kind of underlines the point we don't really understand his problem - but the large organisation that builds these is itself an example of such impossible to understand interactions - maybe we will learn from Azure etc and take those learnings into taking our own orgs.
- throwawaaarrgh 4y ago
  Well anything designed as SOA is just Conway's Law enshrined in architecture. Most older software was too. The org is the problem. The super bizarre outlier is the Open Source community, which doesn't really follow any organizational plan (except the corporate projects and foundations and committees that pretend they're open source). For whatever reason, Open Source succeeds despite no hierarchy, no clearly defined architecture, no particular strategy. But then again, maybe that's why it works so well. An organization has a fixed structure, rules, priorities. It can fail, lose its intellectual property, its funding, be competed against. Open Source has no such limits. It's the wild west. Do whatever you want, whenever you want, and whatever works, survives. Lke natural evolution. An organic ecosystem, rather than a plastic one.

vsareto 4y ago

I wouldn't be surprised if most people don't learn it in-depth because: A) you can google or ChatGPT things more easily these days B) there's a truck load of services and ways to connect and do things. It's hard to learn all of these or pick a best option for everyone C) SSO + identity models + API permissions generally made everything more complex D) Many things in Azure are changing often, so you don't want to get too invested in a particular "version" So things got more in-depth while expanding your available options (which are usually just as complex)

jen729w 4y ago
I think people don’t learn it [all] in-depth because to do so is essentially impossible. Back in the bank days all of understood Windows, multi-tier AD, DNS, DHCP, SMS* (now MECM via SCCM), that distributed file system whose name I forget, file & print, Exchange, and all the other things that had a plugin to mmc.exe — we could do ‘em all. Oh and a handful of networking because that was also much simpler. Oh and all of the client configuration. You could give me a server and a day and I could stand you up a basic infrastructure. I could know it all. (*Though the maniacal way that SMS 2.0 did its magic with text-based log files shuttled in from the endpoint always drove me bonkers.)
- Jupe 4y ago
  And, by the time you "know it all", the playing field has changed. RESTful APIs? Nope, we are going full-gRPC. No hand-rolled OAuth - we are using AuthSolution1. Forget that, AuthSolution1 has a security vulnerability, we are going with AuthSolution2. Wait, AuthSolution2 won't work with cloud providers database offering... let's try AuthSolution3. Let's use IFTTT. No, how about Zapier. But, we need gRPC and now GraphQL. The databases are over-provisioned, lets look at serverless as a critical path item. Executive VP #2 thinks we should use a NoSQL solution. BTW: Cloud provider X is dropping support for database version 1.1, we need to migrate to 2.0... or skip ahead to the just-released 3.0. And, then the CEO comes and asks: "Can you spend some time trying to get ChatGPT to build this whole thing for us?"
  - 4y ago

jiggawatts 4y ago

We might be working on the same project... or the "same" project, if you know what I mean. The level of abstraction that is commonplace these days is insane. At $dayjob, there is a reverse proxy in front of a API gateway that is itself a load-balanced service. There are load-balancers behind it, pointing at a reverse proxy on a service fabric. Within that, some "architect" decided to add additional "mid tier" servers because that was the cool thing to do in the 1990s. Behind all of this is an Azure App Service, which is in turn a load-balanced server that includes a reverse proxy. They want to move this to Kubernetes, so they can now have proxies seven layers deep, nested hypervisors running containers, and services ping-ponging across six zones (data centre locations) in two different clouds. That connectivity will go through a software virtual router platform, so it's not even direct point-to-point connectivity. I pity the poor fool who will have to diagnose an operational issue in this madness. No mere mortal will be able to make it go fast.

userbinator 4y ago
Someone is paying for all of that, and someone is getting paid. Look no further beyond those getting paid to find the reason such monstrosities exist.
- nosianu 4y ago
  No I don't think so. That explanation only works for simple systems, if at all. But all the many players and their very diverse needs and ideas are very complex. I think i's more likely that it's something described in studies about human problem solving, which leads us to preferring overly complex solutions because we prefer adding something. https://www.nature.com/articles/d41586-021-00592-0 https://www.nature.com/articles/d41586-021-00592-0 (you may need Sci-Hub - paywalled) > Adding is favoured over subtracting in problem solving > A series of problem-solving experiments reveal that people are more likely to consider solutions that add features than solutions that remove them, even when removing features is more efficient. https://www.nature.com/articles/s41586-021-03380-y https://www.nature.com/articles/s41586-021-03380-y > People systematically overlook subtractive changes https://www.scientificamerican.com/article/our-brain-typically-overlooks-this-brilliant-problem-solving-strategy/ https://www.scientificamerican.com/article/our-brain-typical... https://www.washingtonpost.com/business/2021/04/16/bias-problem-solving-nature/ https://www.washingtonpost.com/business/2021/04/16/bias-prob... I think we also have forces in play that add to this bias. Or they are the result of it, for some of them. For example, when there is an easy and cheap or even free solution there is no incentive for marketing and selling it. It's hard to tell a customer that they don't actually need you, especially if it isn't a one-off, or that they should go somewhere cheaper and more simple. It is much easier to get paid for adding than for doing nothing or for subtracting, both as an employee and as a business. It certainly looks that way when I look around. We have frequent complaints about a lack of skilled labor - but that's also because everything keeps getting more and more complex. For example, once it was enough to once every half year update a piece of paper on each bus or tram or train station with the schedule. Now it's some highly complex networked computer solution and display to display when the next bus is going to arrive. Sure there is added value - but there also is a gigantic amount of new complexity for every small "convenience". When it comes to something like waiting for the next tram, specially in an urban environment with 10 minute schedule, the benefit of knowing if it's 2 or 7 minutes that you'll have to wait is negligible compared to the effort of providing and maintaining that service. (we have this situation in my town). Of course there is added value for much of it, maybe all. But I see the complexity and size of systems I knew in a much more simple form when I grew up exploding. The problem is, continuously educating new people to know ever more and to be able to handle more and more sophisticated systems has a continuously increasing cost to society too, since we don't live any longer than before. Never mind everything we do, from planning to maintenance, is ever more complex too. In addition, the new system may not be as robust as the more simple ones of the past. It all works as long as it does. If we see another chip crisis for example, and I just read a pretty dark forecast for China/Taiwan, all those solutions that require working world-wide supply chains, here, chips from Taiwan, are a problem.

userbinator 4y ago

These days we use Azure, because Microsoft has fooled us in to thinking that it’s cheaper. All the cloud providers do that. You could say that peoples' judgment has been clouded... More seriously, this is how they operate; complexity is a good thing to them, because it means more lock-in and ways to charge you in non-obvious ways, and the whole "consulting" aspect of the industry feeds off that.