11 ms·

Not pictured: 2005 your average box served some php and static assets, connecting to some generic relational database. Reading logs means grepping files over s

Not pictured: 2005 your average box served some php and static assets, connecting to some generic relational database. Reading logs means grepping files over ssh. 2022 your architecture runs in the cloud, has multiple flavors of databases, queues, caches, and so on. You have at least two orders of magnitude more complexity because you aren’t just serving a web page anymore - you handle payments, integrate with other services, queue tasks for later, and so on. Your automation may be an order of magnitude more complex than 2005, but it enables two orders of magnitude more functionality.

stathibus 4y ago
My classic C programmer curmudgeon take is that the root of the problem, as with everything else in this industry, is bad software built on top of bad software built on top of bad software... and on it goes. The systems in your 2022 world are hard to test and maintain because they are bad, and the tools we built to test and maintain them are largely built on the same foundational ideas and technologies, so they are even worse. We're going to have to rip everything back down to the foundation in order to make progress beyond finger-pointing (X is DevOps but Y is software engineering and Z is IT admin).
- theflyinghorse 4y ago
  What software is bad precisely? The modern browsers? The tools we use to build modern browser apps? Perhaps your gripe is with something else entirely, like modern databases? Or is it the OS that you don't like? Really not sure what you're commenting about here
  - Godel_unicode 4y ago
    They said that the world is bad software on top of bad software, so it’s not one thing. It’s everything being bad and working around everything else being bad.
    - voidfunc 4y ago
      Yea but they didn’t explain why its bad. Software devs complaining about “bad software” is basically an industry trope at this point but often it just means “Its too complex and I don’t understand why that complexity is probably necessary” or “Its not written the way I would have written it”.
    - arinlen 4y ago
      > They said that the world is bad software on top of bad software, so it’s not one thing. I don't know about OP, but I personally feel like software like GCC/clang/MSVC and Debian/Ubuntu and Docker and SSH and Git are pretty great, and were never better. Heck, the whole dotnet ecosystem is turning some significant problems in developer experience into at most minor nuisances. Even Firefox+Chrome are stellar, and extremely solid as-is. Where exactly is this bad software OP is talking about?
      - happymellon 4y ago
        In my experience it's the custom scripts in an attempt to separate infra from developers. If you are using one of the major cloud providers and instead of just using a native IaC scripts, you have decided that it would be better if the developers only had access to your custom kubernetes operators to manage their infrastructure then you are the problem. If you are using a deployment pipeline that developers have zero involvement in, and they just "import your Jenkins scripts" then you are the problem. If you are using major cloud provider and rather than just using their managed kubernetes, you have decided to deploy your own, then the chances are that you are the problem. Rarely have these approaches ever been stable, in my 20+ year experience they have never been stable. And they are the reason that "real DevOps" came along because developers could do a better job with less downtime if people actually engaged them.
  - cshenton 4y ago
    Why not all of the above?
  - metadat 4y ago
    Some people feel anything not written or created by them is "bad". It doesn't matter to me who turned the wrench, but in my career I've encountered these types where it's only good enough if it's theirs. And IMHO their output is generally quite atrocious. Root cause: Psychological issues.
    - hinkley 4y ago
      Shit most of the stuff I write is bad too. What’s missing a lot is empathy, and that takes slowing down and observing.
      - patrick451 4y ago
        We have way too much empathy for bad software. Honestly, it's almost all total shit. If my care was as bad as the median level of software, it would last 30 days and then be broken down in my driveway until the scrapper hauls it off.
        kaba0 4y ago
        What part of the stack you think of? Because my experience is that the underlying stuff Just Works^TM. Sure, it does have the occasional hiccup, but 1 out of 1000, it is some higher level tool that has a bug. Just to share the terminology, under underlying stuff I mean the linux kernel, the JVM, etc, while higher level stuff could be certain build tools (khm, looking at you npm), and of course end user tools that are unfortunately way too buggy. Like I have a hard time listing end user applications where I didn’t notice a bug yet.
        hinkley 4y ago
        I feel like there's an opportunity missed here with the transition to clustering for us to build a small kernel for running simple services and moving things over to run on top of these things. Give me a tight, highly coherent API, with a culture of avoiding feature factory work. At least get "Make it work, make it right, make it fast" to not skip "make it right" anymore. That's sort of the core illness in software today. Open source doesn't have to be pushed by business concerns. There's a degree of regulatory capture, but the biggest problem is that we just don't know any better. We repeat what we see, and make sure our own pain points are handled. I've seen this play out in API design where an awful API is introduced, and each person who fixes it only fixes 20% of the pain, and so it's 10 years before we get from a mediocre library to one you could actually call 'good', because nobody made 'good', they made better than better than better than ridiculous.
- phillipcarter 4y ago
  Talk about a low-effort take on the state of modern software. You know what’s also “just bad”? Most systems software written in C.
  - stathibus 4y ago
    We could do a case study or something, but that would be a blog post or a book, not an HN comment. How much effort are you looking for here?
  - no-dr-onboard 4y ago
    I was put off by the low effort take as well. Personally, I’ve worked through having this mindset myself. I “grew” up with the C and tots descendant family of languages. I wrote shellcode in C and ASM as an exploit dev. I later learned python and now ruby, arguably some of the most abstracted languages. At each iteration I openly opined at how “C would let me do X without the handcuffs I put myself in by using Language Y”. It’s really just senseless complaining. There is a reason why someone chose Ruby over PHP. PHP over pascal, Pascal over Ada and so on. The point is, to scoff at the product as a bystander is just immature. Throwing shade on an entire industry based off some weird pet purist outlook is just immature :/. Of course, the irony here is that this type of mindset is no different from the apprentice carpenter who scoffs at every new house he walks through that he didn’t build. It’s no different from the wine snob who cringes at what his mom puts out when company is over. It’s no different from the tens of submissions HN accumulated in a week with something of the tune “I rewrote X in Rust”.
- anothernewdude 4y ago
  Expecting all software in a properly written stack to be "good" is a particularly bad take, and one that leads to brittle software. Best to accept reality, deal with it and move on.
- bitwize 4y ago
  Your classic C programmer take comes from a time and place where servers were pets and nobody had to, or could, scale very big. The fact is that businesses either are or are converting to cloud-first because why incur the risks of maintaining the one special-snowflake server that runs your business when you can turn a crank and spin up as many instances as you need -- theoretically very many as you take on more customers and transactions and need to handle the load? And ypu can just restart them if they go down? Stop thinking technology and start thinking BUSINESS. The cloud may be more janky and complicated than what you're used to but it serves the business's needs better.
  - jacquesm 4y ago
    Lots of companies excel at playing big business without actually being a big business (or having a remote chance of becoming one). They simply lug around the kind of infrastructure that might be suitable for a company ten times their size. Ironically, it isn't rare that that extra infrastructure and the cost and effort to maintain it limit them in the marketplace.
  - lelanthran 4y ago
    > The fact is that businesses either are or are converting to cloud-first because why incur the risks of maintaining the one special-snowflake server that runs your business when you can turn a crank and spin up as many instances as you need -- theoretically very many as you take on more customers and transactions and need to handle the load? And ypu can just restart them if they go down? I'm pretty certain you can do the same with software written in C (or any other language) without resorting to AWS/Azure/GCP/etc. You can do the same cranking on your own hardware, except you keep the hardware after the peak has passed and you need to plan for the peak. Or, you can simply get a DO droplet, and crank out as many new instances as you need, and turn them off when you don't, with software written in C (or anything else). There's no need to go all-in on the AWS/Azure/GCP solution, with the ability to orchestrate instances as cattle, when you have, at peak, 3 instances[1]. [1]If you wrote your software in C, or Rust, or Go instead of Python, or Ruby, you could potentially get away with fewer instances than you need, thereby reducing to treating the system as a system of pets because there are so few of them.
- vinyl7 4y ago
  The reason is this: To make money in a gold rush, sell shovels. Everyone wants to bank on the software craze, and so everyone develops pointless middleware solutions.
  - drpyser22 4y ago
    That's a bit disingenuous to actual engineers trying to solve problems and genuinely believing their solution can make things better. Also, looking at principles over tooling, DevOps has a lot to offer that trivially make sense.
  - pjmlp 4y ago
    You're forgetting the conferences, books, trainings, consulting gigs.
- deleted 4y ago
  [deleted]
- posnet 4y ago
  You are not alone in thinking that, see 'The Thirty Million Line' talk by Casey Muratori. (Long, but worth it, assuming you haven't seen it already) https://www.youtube.com/watch?v=kZRE7HIO3vk https://www.youtube.com/watch?v=kZRE7HIO3vk
- dasz 4y ago
  I don't think we need to muddy the water by saying bad software. Just write the word software. It's multiple layers upon layers. This isn't automatically bad. There's so much more going on. I don't know if the complexity is all required but we're doing more now. There's just more going on. That's how it is. We need the abstractions. Maybe not all of them. But we need more abstractions and tools running now. There's so much to manage. I don't see us reducing this complexity that much other than consolidation and some simplification to clean things as they get sorted further. But there is still a need for the layers. It's not 2005. Things are more complex. The nearest analogy is would you like to code everything in line numbered BASIC or would you like to do it using some modern equivalent? Even with the additional layers there has been real progress in the tools. Improvements across a multitude of metrics. It's not all complexity for the sake of complexity. Smile. It's another stage of progress. The old mess will fall away. At the next stage what we see as an improvement now will be the next stage's mess. There's a whole slew of problems that will need even newer tools. We haven't even scratched the surface of the automation we'll require in ten years time.
- oblio 4y ago
  > We're going to have to rip everything back down to the foundation in order to make progress beyond finger-pointing (X is DevOps but Y is software engineering and Z is IT admin). 1. Never going to happen unless everything collapses, which is extremely unlikely. 2. We are progressing. AR, VR, GPUs with raytracing, complex web applications and global services, global cloud, etc.
- 1vuio0pswjnm7 4y ago
  To whom does "we" refer. The classic C programmer curmudgeon. Those commenting on HN frequently use the term "we" but reading HN one can see that, amongst those commenting, there is a divergence of opinions about software. Who is "we". Asking prospective "DevOps", "software engineer" and "IT admin" candidates to program in C is an effective way to weed out those who are not truly capable of writing "good" software. It is easy to find mistakes and expose the incompetent. That may offend the incompetent who believe they are "good" programmers. Thus there are new programming languages created every year, more people using them to write "bad" software, and more people who feel emboldened to attack C as the source of problems, instead of "bad" programmers. C is truth serum. It is difficult for people programming in C to pretend they are "good" programmmers. They will be found out. It is funny how people on HN attack the language for exposing so many "bad" programmers. Under this theory, programmers are absolved of all responsibility for their mistakes. The language is at fault. "Good" programmers do not blame languages for their own mistakes. IMHO, there is sometimes a benefit to languages that make it more difficult, not easier, to create and sustain Rube Goldberg complexity. Not to mention languages that compile to smaller, faster programs. "Programmer productivity" is ambiguous. For example, it could be a positive for a programmer trying to justify the expense their salary presents to an employer or it could be a negative to people who are forced to use an ever-increasing quantity of "bad" software. It could be a positive to a programmer who wants to keep implementing new "features" or it could be a negative to a software user who dislikes "feature creep". The dichotomy of good versus bad software is such a subjective topic that the term "we" really needs to be defined. Different groups of people have different interests and therefore different opinions.
  - mlyle 4y ago
    Hey, I've written more C code than anything else and was generally considered pretty competent. > The language is at fault. "Good" programmers do not blame languages for their mistakes. I also know that my time to get something functional in higher level languages is often 10x less, and my probability of having very subtle bugs to hunt is much lower. There's a super-weird tradeoff here. There's all kinds of modern, high-level techniques that improve programmer productivity and reliability of bigger systems.. They convert people who can't be productive C programmers into doing OK work. But they're also slow and a bit opaque in how they work. And it's really easy to run out of performance and have to do exotic things to get them to scale, in which case you have to build bigger systems still and cede a lot of those advantages. Whereas, if you write to the metal, you can get a lot of performance out of a single large computer.
- arinlen 4y ago
  > My classic C programmer curmudgeon take is that the root of the problem, as with everything else in this industry, is bad software built on top of bad software built on top of bad software... and on it goes. No, not really. Nowadays things are way way better than how they were a decade ago. Nowadays you can get pipelines that build and test on multiple target platforms with a dozen or so lines of code, and things run so well that they became reliable infrastructure integrated with your prod system. Back then you had none of this, and had to pay the salary for a "build engineer" who worked like a wizard stirring a cauldron just to get a single production build out of the door.
  - blueflow 4y ago
    The bad things are still around in your infra even if you manage to never see them because you only see the "dozens lines of code" at the front.
    - mrjin 4y ago
      Someone who really buys that "dozens lines of code" will definitely has no idea what kind of complexity it was about.
  - hinkley 4y ago
    I’ve been cultivating this opinion that it’s the other way around. In any other domain, few people use crap tools to make something amazing, and I think it comes down to what you surround yourself with informs what you make. We need better tools if we want better things.
  - mrjin 4y ago
    Oh sure. Can you try to set up ANY CI pipelines to make it more or less working, don't worry about "pipelines that build and test on multiple target platforms" and get back to share with us your experiences?
    - allarm 4y ago
      What exactly do you want to hear? Not op, but I’ve been doing it for some time.
Sebb767 4y ago
The big question is: Do you need that kind of functionality? I agree that very large and complex infrastructures have their place - the problem is just that they import a ton of complexity and usually cost a lot. People are always suprised when they see a minimal webserver instance on a ten year old debian [0] handling tens of thousands of requests, without issue. It might go down once a decade because the disk failed, but it cost 1200$ to run over that decade. I don't think that's the perfect way, but modern infrastructures love to include a lot of complexity when it's not needed. The problem is that including something is very easy and the cost is only payed once it breaks. Also, hardware is really cheap (you might not think so, but if you compare it to western IT salaries, it is). [0] I know outdated instances with no update strategy are not really a benchmark, but I encourage you to go out and ask people what their container base image upgrade strategy is - the situation really did not change that much.
- _vertigo 4y ago
  Kind of depends on what you’re building, right? Avoiding building new features in order to limit complexity is probably a good idea in select cases but not in others. Using more complex, purpose-built technology over standard boilerplate is often very cost-efficient, but only if you run at a scale where the additional complexity is worth the savings…
- fipar 4y ago
  > The big question is: Do you need that kind of functionality? I agree that very large and complex infrastructures have their place - the problem is just that they import a ton of complexity and usually cost a lot. I think that's the key. Most of us aren't Google/Facebook/whatever. Yet lots of people want to mimic those. A simple stack with a relation database on a single machine still goes a long way. In fact, it goes a lot longer way today than it did 20 years ago. It's ok to scale when you need to, but I think premature scaling is the wrong way to go, especially since most things never get to FB/Google/etc size.
  - bluepizza 4y ago
    I understand trying to keep complexity low. But I don’t understand how a relational database would be less complex than a simple key value store. Every time this argument pops up, I end up wondering if it’s really about architectural complexity, or if it’s about sticking with the old and known.
deleted 4y ago
[deleted]
terpans 4y ago
I was working on large scale deployments with multiple datacenters, multiple databases, message passing networks and running multiple customer-facing products. Load balancers and HA setups were already in use. LVS existed. VMs were popular. "CI/CD" existed, without that name. > 2005 your average box ... Speak for yourself.
- _vertigo 4y ago
  - terpans 4y ago
    Indeed all of those things existed and they weren’t in widespread use because 99% of organization did not need them. And it hasn't changed. > 2 sysadmins were absolutely not managing “large scale deployments with multiple data centers” Please don't make things up because I was there. > But today it’s actually super plausible that 2 or 3 Devops with modern tooling can manage all of that. Not really.
    - _vertigo 4y ago
      - deleted 4y ago
        [deleted]
      - dijit 4y ago
        I was there too and some things were better, some thing were worse. The problem is that we’re being sold a lie and it’s hard to swallow. “Cloud will save you on staffing costs” - no, it just means you have specialists working on proprietary solutions, just like IBM mainframes or storage appliances or load-balancers/traffic-shapers of yore. “This technology will make rollouts easier” - until it breaks down and is not easy to put together again, you’re at the mercy of your upstream and you had better hope you keep a really solid update cadence and not introduce something you can’t consume later. “Layers on layers means things are composable” - sometimes, but you need more people to know more things to put it together properly. Running everything from QuickStarts is doomed to fail, but I see so much of it. Our config management tools back in the day were crummy as hell, cfengine being the only notable one (which was awful), our monitoring systems required a lot more handholding and truthfully: people were not happy to talk through requirements; which was probably why “devs run production” was appealing. these things have gotten better, but nearly everything else has definitely gotten worse.
thezilch 4y ago
Classic over-engineered DevOps, adding complexity for complexity sake. When you have a hammer... Everything you described was 2005.
- HelloNurse 4y ago
  In 2005 there were, for instance, fancy J2EE application servers instead of fancy container roulette tools.
rco8786 4y ago
The fun part is that the 2005 architecture is still plenty sufficient for 99% of deployments but everybody architects for 100x the scale they actually need.
- tech_tuna 4y ago
  This is a good point, but I'd say that's just the ops version of premature optimization.
  - int_19h 4y ago
    No, it's the ops version of premature abstraction - when devs build a whole extensible framework around a simple use case just because "requirements might change later" (which 90% of the time they never do, so on average the work is wasted much more often than not).
    - tech_tuna 4y ago
      I mean yes. . . premature abstraction driven by premature optimization. Or more simply, solving a problem you don't have now, and might not ever have. Everyone talks FAANG scale on day 1. It's common and absurd.
- bottled_poe 4y ago
  Usually because most business say they need highly available, scalable infra from day 1.
  - rco8786 4y ago
    I typically see it happen in the other direction myself. Business folks describe the problem/product and engineering team says “oh yea we can do that, we’ll need K8s” when they do not, in fact, need K8s
tech_tuna 4y ago
This times one million. Back in 2005 there would have been so much shit you wouldn't even dream of doing and a lot of things you did do, you sure as shit wouldn't do now. . . we used to write our cache systems. Migrations were all custom scripts. Fucking nightly builds were a thing because we didn't kick of builds on commit. Unit tests weren't even common practice back then. Yeah, most places had tests but there was no common language to describe them. And as much as git can be a big complex pain. . . merging was a BIG thing back then too. I seldom deal with long lived branches and the nightmarish merges they often needed. Also, to all the young folks who "want to work with physical servers" again. Have fun with the next set of device driver patches you need to roll out. I heart my containerized IaC buzzword laden cloud existence. #FuckThatNoiseOps
- raffraffraff 4y ago
  It's as much a cultural problem as a 2005 vs now problem because I have experience working at a joint that uses AWS yet chooses to roll their own Kubernetes, uses GitHub yet flails around with long lived branches and 20 minutes CI stages, uses containers but still has 20 minutes build times, uses ArgoCD yet has tiresome "release events" that generally can't be rolled back and take at least an hour to "fix-fowards" after a bad release. Sometimes you can lead a horse to water and even push it into the trough, but it would rather eat dust than let go of its last-decade ideas about operations.
mbesto 4y ago
> Your automation may be an order of magnitude more complex than 2005, but it enables two orders of magnitude more functionality. And can create more than 2 orders of magnitude in terms of economic value (i.e. profit) as a result.
- lkrubner 4y ago
  If that was true, why has the economy since 2000 never hit the speed of growth that it had in the 1990s? It seems the big economic gains of the Internet came early, and what we've seen since is the Law Of Diminishing Returns: increasing inputs for decreasing outputs. Bigger investments for less economic growth.
  - mbesto 4y ago
    > hit the speed of growth that it had in the 1990s? It seems the big economic gains of the Internet came early It did, just not in the way you're describing it. This is how it's manifested itself: > Apple, Amazon, Alphabet, Microsoft and Facebook all account for just under 20 percent of the market value for the entire S&P 500. With a collective value of nearly $5 trillion, these top tech companies easily dwarf other entire industries in the index, with companies like Berkshire Hathaway and JPMorgan Chase falling well short. Currently, the total valuation of the S&P 500 is almost $27 trillion. [0] - https://www.statista.com/chart/20794/tech-companies-highly-valued-in-sp-500/ https://www.statista.com/chart/20794/tech-companies-highly-v...
    - lkrubner 4y ago
      That is not economic growth. A valuation of some companies on the stock market has noting to do with economic growth. "Create more than 2 orders of magnitude in terms of economic value" never actually happened.
      - mbesto 4y ago
        > 2 orders of magnitude in terms of economic value (i.e. profit) Then you clearly misread my initial point. economic value != overall economy growth
onion2k 4y ago
2005 Your app takes some user data, stores it in a database, and presents a filtered view of it to the user on request. 2022 Your app takes some user data, stores it in a database, and presents a filtered view of it to the user on request.
- _vertigo 4y ago
  Well if that’s all you’re doing in 2022, there’s nothing wrong with deploying your code and grepping logs over ssh. Bash scripts and offloading some of the complexity of management to AWS (e.g. use autoscaling) goes pretty far
  - HelloNurse 4y ago
    Autoscaling? Do you mean facing so much cloud management complexity that it invites automation, in order to spend more money for a herd of excitingly ephemeral server instances than for a reasonably overprovisioned boring actual computer?
  - onion2k 4y ago
    In 2002 I built a website that got about 3m page views a week, with about 50k concurrent users, and a whole bunch of fun things like payments, outbound email, etc. It ran on a single server. Since then I've working on things that use several servers, things that use no servers with edge functions instead (which have servers, but still), and that use autoscaling across lots of servers. No doubt some businesses need autoscaling but most don't. It definitely shouldn't be something you reach for before you actually anticipate a need for it.
fipar 4y ago
I partially agree with what you're saying but let's not pretend we weren't handling payments in 2005, some of us where anyway. I think what changed is the scale of things: we had a lot less people online back then. I think the increased complexity in our architectures is correlated with the DevOps role coming to the picture, but I'm not sure there's a cause link there. My recollection from having lived through the initial years of DevOps (I began working professionally in 2000) is that an important goal was to reduce the terrible friction between dev and ops (the latter including sysadmins, DBAs, etc). Whatever the extra complexity we have now, I would not want to go back to the days when I carried a pager and responded to find out the problem was caused by an application I had no insight into, and there was no corresponding on call person on the side of the dev team. Another important goal was to manage infrastructure a bit closer to how we develop software, including the big step of having the configuration (or the code that generates it) in SCM. Another thing I don't miss is logging into a server to troubleshoot something and finding /etc/my.cnf, /etc/my.cnf.<some date>, /etc/my.cnf.test, etc. It's much easier to just run tig on either the file or the ansible/chef/whatever that generates it IMHO.
democracy 4y ago
2005 was fullon enterprise websphere/weblogic era, so no, if anything it was much more complex (from architectural side) than today's python/nodejs or even spring boot solutions. Automation (bash/ant) plus ci like cruise control already there.
littlestymaar 4y ago
> you handle payments, Yeah, because obviously nobody handled any payment in 2005 … The same can be said for everything in your list. And more importantly, it's unlikely that your business is more complex than what it used to be in 2005. And if you're spending more resource to deliver the same business value, you're wasting them. (That's why PHP is still ubiquitous btw, it sucks by most standards, but it's good enough to do business and that's what matters).
Jistern 4y ago
>> Your automation may be an order of magnitude more complex than 2005, but it enables two orders of magnitude more functionality. This! The primary problems with DevOps are... 1. It is still in its infancy therefore it's changing quickly. 2. Bad (or no) documentation is much more painful than before. A single family house without blueprints can, usually, be adequately serviced; whereas, a 27 story office building cannot.
arinlen 4y ago
Also not pictured: * 2005: ship it if it builds. * 2022: run unit testing, deployment to beta, run integration testing, run UI testing, deployment to preprod, run internationalization testing, run regional/marketplace-specific testing, run consistency checks, run performance tests, deployment to prod.
- takeda 4y ago
  I think the variable here isn't year, but where you work(ed).
  - kitd 4y ago
    Maybe, but the pipeline required to do all that in 2005 would be considerably more complex and brittle than the 2022 version using tools and infra designed for the job.
deleted 4y ago
[deleted]
dhzhzjsbevs 4y ago
Missing the point: That code in 2005 was better in every way. Context: just learning loopback API framework. What a steaming pile of overengineered garbage.
hsn915 4y ago
All you did is describe the increase in complexity of the infrastructure. Which does not rebut the point, because that is the point. The infrastructure people are using now is much more complicated than what was common 15 years ago. If you want to rebut this you need to demonstrate the kind of capabilities we have now thanks to this complexity that we could not have before. > you handle payments No you don't. Most people handle payments by integrating with a 3rd party service, most likely stripe or paypal.
- LeonB 4y ago
  Good point re payments. Your system today doesn’t even handle left pad, outsourcing even that to a non deterministic tree of geo-distributed modules.
wruza 4y ago
you handle payments, integrate with other services, queue tasks for later, and so on This just means “use APIs”, adds {base_url, secret_key} pairs to a config file. Where are orders of magnitude devops-wise?
- barking_biscuit 4y ago
  Integrating with other services makes your software more brittle as there are more points of failure from hardware and networking on your end, to API gateways, networking, and hardware on their end. Periodically those things tend to go sideways which means whatever it was your system was trying to do doesn't happen, or worse it only partially happens. Usually some poor engineer is then tasked with figuring out if this is an on-going problem or a one-off problem? Did it affect a single customer, or is it affecting all customers? Is it affecting a single instance, or is the problem happening across all instances? Is it a problem on our end, or is the 3rd party API misbehaving (again) today? Generally you need solid APM and logging to support the operations of this. Then there's queueing tasks for later and making sure you never lose messages between systems. All sorts of fun happens when you hit a capacity limit in your queueing system due to unintended/unnoticed design flaw and all of a sudden system performance starts to tank as things aren't getting processed due to too much stuff in the queue that isn't draining at a sufficient rate. You need to figure out is the queue actually draining or is it still increasing in size? Are all the worker nodes still running or did one (or several) of them hang and didn't die but simply isn't processing any jobs out of the queue but still consuming a spot in your auto-scaling group blocking a new healthy instance from coming online? Is it a problem in your software or is it actually a blip in operations from your cloud-provider? It's way, way, way more complex than simply "chuck some API keys in a file and she'll be right!". That's hobby project level stuff, not realistic ops of a production system.
  - wruza 4y ago
    I’m literally working at automated finance and we are offloading these cases to client support, and late/next day status requests for b2b peers. The only devops-level technicality there is an in-process keep-alive system which prevents overwhelming the support repeatedly. What you’re describing tries to deal with these issues at the level where it is hard, and it is, but that’s exactly the road to complexity and infra costs (and infra issues as well). As a result, it trades simple client issues for hard technical ones, and you still need support. It’s cool to have an ideal incident-free system, but only when you can cover its costs - and more importantly its evolution barriers - by exactly that quality. Iow, don’t replace humans with machines either until machines learn to wipe their own ass, or until it’s just too much ass to manage efficiently anyway. I’d fight at your side few years ago, but since starting to work in this field and asking around I realized nobody cares about that ideal way, cause it’s so expensive and hard to maintain, and the cost of change is prohibitive. It even left few scars on my mental health, exactly because I was too anxious to go this way after rotating in environments where devops is praised as something inevitable to survive. If you squint at us at the right angle you still can see devops, but I think we are just expressing two opposite sets of ideas, each applicable in its own type of a business env. You may still call that a hobby-level, and I’d even agree (cause it essentially is) if that hobby didn’t bring as much as to sustain and profit the company at the levels many companies would wish they were at. If it works and brings revenue who cares how it’s categorized.
xkbarkar 4y ago
Wanted to add, todays pipelines include a whole new level of security that was pretty much ignored in 2005. Security adds complexity
- terpans 4y ago
  > todays pipelines include a whole new level of security that was pretty much ignored in 2005. The very opposite. > Security adds complexity If anything, complexity is the enemy of security
GrumpyNl 4y ago
The question is, do you need all that or was/is PhP and Mysql enough for the job.
HelloNurse 4y ago
Extremely complicated architectures are a liability, not a feature to be proud of; increasing complication (i.e. costs) doesn't mean increasing functionality (i.e. value). For example, why do you "queue tasks for later"? Do you have exceptionally punishing latency and throughput requirements that can only be met at a reasonable cost by not answering synchronously, or it's because your database doesn't do transactions? Similarly, what do you do with "queues, caches, and so on"? Meet further extreme performance and availability requirements, or attempt to mask with additional complications the poor performance of inefficient complicated components? In 2005, but also in 2000, web applications had already progressed past "just serving a web page", mostly without complicated architectures and therefore without the accompanying tools. I think tool improvements made cargo-culting actually advanced state of the art software architectures and processes easy and affordable, creating a long term feedback loop between unnecessary demand for complexity (amateurs dreaming of "scaling up") and unnecessary development and marketing of advanced (but not necessarily good) complexity-management tools, often by the same amateurs.
tsarchitect 4y ago
Also not pictured-- 2005: two system administrators wrote all automation using a handful of Bash, Perl and Python scripts AND LEAVE THE COMPANY a couple of years laters 20xx: new hired system administrators continue to rewrite scrips. no shared knowledge because scortched-earth policy in effect 2022: HN: DevOps is a failure -- you just need two system administrators...
phendrenad2 4y ago
> you handle payments, integrate with other services, queue tasks for later I distinctly remember buying things on Amazon in 2005. Actually come to think of it, I did everything you list in 2005, at multiple companies.
bayindirh 4y ago
I find this take apologetic. "But we're running on the cloud, we need this complexity", is a wrong take. Yes, the underlying services may need this, but the infrastructure managing these services don't need to be this complex. Every complex tool can be configured to work in a simpler manner, and all configuration repositories directing these tools can be organized much better. It's akin to a codebase after all. Many code quality processes and best practices to apply to these, but sysadmins don't like to think like developers, and things get hairy (I'm a sysadmin who does development, I see both sides of the fence). The sad part is, these allegedly more robust platforms do not provide the transparency neither to developers, nor to sysadmins, nor to users. In the name of faster development, logging/debugging gets awry, management becomes more complex and users can't find the features they have used to have. Why? We decoupled systems and decided to add this most used feature at a later date, because it's not something making money. Now my account page even doesn't show my e-mail address, I can't change my subscription renewal date, or see my past purchases, why? They're at distant tables or these features are managed by other picoservices which are not worthy to extend or add more communication channels. Result? Allegedly mature web applications with weird latencies and awkwardly barren pages, with no user preferences or details. Adding layers and layers of complexity to patch/hide shortcomings of your architecture doesn't warrant a more complex toolset to manage it.

stathibus 4y ago

My classic C programmer curmudgeon take is that the root of the problem, as with everything else in this industry, is bad software built on top of bad software built on top of bad software... and on it goes. The systems in your 2022 world are hard to test and maintain because they are bad, and the tools we built to test and maintain them are largely built on the same foundational ideas and technologies, so they are even worse. We're going to have to rip everything back down to the foundation in order to make progress beyond finger-pointing (X is DevOps but Y is software engineering and Z is IT admin).

theflyinghorse 4y ago
What software is bad precisely? The modern browsers? The tools we use to build modern browser apps? Perhaps your gripe is with something else entirely, like modern databases? Or is it the OS that you don't like? Really not sure what you're commenting about here
- Godel_unicode 4y ago
  They said that the world is bad software on top of bad software, so it’s not one thing. It’s everything being bad and working around everything else being bad.
  - voidfunc 4y ago
    Yea but they didn’t explain why its bad. Software devs complaining about “bad software” is basically an industry trope at this point but often it just means “Its too complex and I don’t understand why that complexity is probably necessary” or “Its not written the way I would have written it”.
  - arinlen 4y ago
    > They said that the world is bad software on top of bad software, so it’s not one thing. I don't know about OP, but I personally feel like software like GCC/clang/MSVC and Debian/Ubuntu and Docker and SSH and Git are pretty great, and were never better. Heck, the whole dotnet ecosystem is turning some significant problems in developer experience into at most minor nuisances. Even Firefox+Chrome are stellar, and extremely solid as-is. Where exactly is this bad software OP is talking about?
    - happymellon 4y ago
      In my experience it's the custom scripts in an attempt to separate infra from developers. If you are using one of the major cloud providers and instead of just using a native IaC scripts, you have decided that it would be better if the developers only had access to your custom kubernetes operators to manage their infrastructure then you are the problem. If you are using a deployment pipeline that developers have zero involvement in, and they just "import your Jenkins scripts" then you are the problem. If you are using major cloud provider and rather than just using their managed kubernetes, you have decided to deploy your own, then the chances are that you are the problem. Rarely have these approaches ever been stable, in my 20+ year experience they have never been stable. And they are the reason that "real DevOps" came along because developers could do a better job with less downtime if people actually engaged them.
- cshenton 4y ago
  Why not all of the above?
- metadat 4y ago
  Some people feel anything not written or created by them is "bad". It doesn't matter to me who turned the wrench, but in my career I've encountered these types where it's only good enough if it's theirs. And IMHO their output is generally quite atrocious. Root cause: Psychological issues.
  - hinkley 4y ago
    Shit most of the stuff I write is bad too. What’s missing a lot is empathy, and that takes slowing down and observing.
    - patrick451 4y ago
      We have way too much empathy for bad software. Honestly, it's almost all total shit. If my care was as bad as the median level of software, it would last 30 days and then be broken down in my driveway until the scrapper hauls it off.
      - kaba0 4y ago
        What part of the stack you think of? Because my experience is that the underlying stuff Just Works^TM. Sure, it does have the occasional hiccup, but 1 out of 1000, it is some higher level tool that has a bug. Just to share the terminology, under underlying stuff I mean the linux kernel, the JVM, etc, while higher level stuff could be certain build tools (khm, looking at you npm), and of course end user tools that are unfortunately way too buggy. Like I have a hard time listing end user applications where I didn’t notice a bug yet.
      - hinkley 4y ago
        I feel like there's an opportunity missed here with the transition to clustering for us to build a small kernel for running simple services and moving things over to run on top of these things. Give me a tight, highly coherent API, with a culture of avoiding feature factory work. At least get "Make it work, make it right, make it fast" to not skip "make it right" anymore. That's sort of the core illness in software today. Open source doesn't have to be pushed by business concerns. There's a degree of regulatory capture, but the biggest problem is that we just don't know any better. We repeat what we see, and make sure our own pain points are handled. I've seen this play out in API design where an awful API is introduced, and each person who fixes it only fixes 20% of the pain, and so it's 10 years before we get from a mediocre library to one you could actually call 'good', because nobody made 'good', they made better than better than better than ridiculous.
phillipcarter 4y ago
Talk about a low-effort take on the state of modern software. You know what’s also “just bad”? Most systems software written in C.
- stathibus 4y ago
  We could do a case study or something, but that would be a blog post or a book, not an HN comment. How much effort are you looking for here?
- no-dr-onboard 4y ago
  I was put off by the low effort take as well. Personally, I’ve worked through having this mindset myself. I “grew” up with the C and tots descendant family of languages. I wrote shellcode in C and ASM as an exploit dev. I later learned python and now ruby, arguably some of the most abstracted languages. At each iteration I openly opined at how “C would let me do X without the handcuffs I put myself in by using Language Y”. It’s really just senseless complaining. There is a reason why someone chose Ruby over PHP. PHP over pascal, Pascal over Ada and so on. The point is, to scoff at the product as a bystander is just immature. Throwing shade on an entire industry based off some weird pet purist outlook is just immature :/. Of course, the irony here is that this type of mindset is no different from the apprentice carpenter who scoffs at every new house he walks through that he didn’t build. It’s no different from the wine snob who cringes at what his mom puts out when company is over. It’s no different from the tens of submissions HN accumulated in a week with something of the tune “I rewrote X in Rust”.
anothernewdude 4y ago
Expecting all software in a properly written stack to be "good" is a particularly bad take, and one that leads to brittle software. Best to accept reality, deal with it and move on.
bitwize 4y ago
Your classic C programmer take comes from a time and place where servers were pets and nobody had to, or could, scale very big. The fact is that businesses either are or are converting to cloud-first because why incur the risks of maintaining the one special-snowflake server that runs your business when you can turn a crank and spin up as many instances as you need -- theoretically very many as you take on more customers and transactions and need to handle the load? And ypu can just restart them if they go down? Stop thinking technology and start thinking BUSINESS. The cloud may be more janky and complicated than what you're used to but it serves the business's needs better.
- jacquesm 4y ago
  Lots of companies excel at playing big business without actually being a big business (or having a remote chance of becoming one). They simply lug around the kind of infrastructure that might be suitable for a company ten times their size. Ironically, it isn't rare that that extra infrastructure and the cost and effort to maintain it limit them in the marketplace.
- lelanthran 4y ago
  > The fact is that businesses either are or are converting to cloud-first because why incur the risks of maintaining the one special-snowflake server that runs your business when you can turn a crank and spin up as many instances as you need -- theoretically very many as you take on more customers and transactions and need to handle the load? And ypu can just restart them if they go down? I'm pretty certain you can do the same with software written in C (or any other language) without resorting to AWS/Azure/GCP/etc. You can do the same cranking on your own hardware, except you keep the hardware after the peak has passed and you need to plan for the peak. Or, you can simply get a DO droplet, and crank out as many new instances as you need, and turn them off when you don't, with software written in C (or anything else). There's no need to go all-in on the AWS/Azure/GCP solution, with the ability to orchestrate instances as cattle, when you have, at peak, 3 instances[1]. [1]If you wrote your software in C, or Rust, or Go instead of Python, or Ruby, you could potentially get away with fewer instances than you need, thereby reducing to treating the system as a system of pets because there are so few of them.
vinyl7 4y ago
The reason is this: To make money in a gold rush, sell shovels. Everyone wants to bank on the software craze, and so everyone develops pointless middleware solutions.
- drpyser22 4y ago
  That's a bit disingenuous to actual engineers trying to solve problems and genuinely believing their solution can make things better. Also, looking at principles over tooling, DevOps has a lot to offer that trivially make sense.
- pjmlp 4y ago
  You're forgetting the conferences, books, trainings, consulting gigs.
deleted 4y ago
[deleted]
posnet 4y ago
You are not alone in thinking that, see 'The Thirty Million Line' talk by Casey Muratori. (Long, but worth it, assuming you haven't seen it already) https://www.youtube.com/watch?v=kZRE7HIO3vk https://www.youtube.com/watch?v=kZRE7HIO3vk
dasz 4y ago
I don't think we need to muddy the water by saying bad software. Just write the word software. It's multiple layers upon layers. This isn't automatically bad. There's so much more going on. I don't know if the complexity is all required but we're doing more now. There's just more going on. That's how it is. We need the abstractions. Maybe not all of them. But we need more abstractions and tools running now. There's so much to manage. I don't see us reducing this complexity that much other than consolidation and some simplification to clean things as they get sorted further. But there is still a need for the layers. It's not 2005. Things are more complex. The nearest analogy is would you like to code everything in line numbered BASIC or would you like to do it using some modern equivalent? Even with the additional layers there has been real progress in the tools. Improvements across a multitude of metrics. It's not all complexity for the sake of complexity. Smile. It's another stage of progress. The old mess will fall away. At the next stage what we see as an improvement now will be the next stage's mess. There's a whole slew of problems that will need even newer tools. We haven't even scratched the surface of the automation we'll require in ten years time.
oblio 4y ago
> We're going to have to rip everything back down to the foundation in order to make progress beyond finger-pointing (X is DevOps but Y is software engineering and Z is IT admin). 1. Never going to happen unless everything collapses, which is extremely unlikely. 2. We are progressing. AR, VR, GPUs with raytracing, complex web applications and global services, global cloud, etc.
1vuio0pswjnm7 4y ago
To whom does "we" refer. The classic C programmer curmudgeon. Those commenting on HN frequently use the term "we" but reading HN one can see that, amongst those commenting, there is a divergence of opinions about software. Who is "we". Asking prospective "DevOps", "software engineer" and "IT admin" candidates to program in C is an effective way to weed out those who are not truly capable of writing "good" software. It is easy to find mistakes and expose the incompetent. That may offend the incompetent who believe they are "good" programmers. Thus there are new programming languages created every year, more people using them to write "bad" software, and more people who feel emboldened to attack C as the source of problems, instead of "bad" programmers. C is truth serum. It is difficult for people programming in C to pretend they are "good" programmmers. They will be found out. It is funny how people on HN attack the language for exposing so many "bad" programmers. Under this theory, programmers are absolved of all responsibility for their mistakes. The language is at fault. "Good" programmers do not blame languages for their own mistakes. IMHO, there is sometimes a benefit to languages that make it more difficult, not easier, to create and sustain Rube Goldberg complexity. Not to mention languages that compile to smaller, faster programs. "Programmer productivity" is ambiguous. For example, it could be a positive for a programmer trying to justify the expense their salary presents to an employer or it could be a negative to people who are forced to use an ever-increasing quantity of "bad" software. It could be a positive to a programmer who wants to keep implementing new "features" or it could be a negative to a software user who dislikes "feature creep". The dichotomy of good versus bad software is such a subjective topic that the term "we" really needs to be defined. Different groups of people have different interests and therefore different opinions.
- mlyle 4y ago
  Hey, I've written more C code than anything else and was generally considered pretty competent. > The language is at fault. "Good" programmers do not blame languages for their mistakes. I also know that my time to get something functional in higher level languages is often 10x less, and my probability of having very subtle bugs to hunt is much lower. There's a super-weird tradeoff here. There's all kinds of modern, high-level techniques that improve programmer productivity and reliability of bigger systems.. They convert people who can't be productive C programmers into doing OK work. But they're also slow and a bit opaque in how they work. And it's really easy to run out of performance and have to do exotic things to get them to scale, in which case you have to build bigger systems still and cede a lot of those advantages. Whereas, if you write to the metal, you can get a lot of performance out of a single large computer.
arinlen 4y ago
> My classic C programmer curmudgeon take is that the root of the problem, as with everything else in this industry, is bad software built on top of bad software built on top of bad software... and on it goes. No, not really. Nowadays things are way way better than how they were a decade ago. Nowadays you can get pipelines that build and test on multiple target platforms with a dozen or so lines of code, and things run so well that they became reliable infrastructure integrated with your prod system. Back then you had none of this, and had to pay the salary for a "build engineer" who worked like a wizard stirring a cauldron just to get a single production build out of the door.
- blueflow 4y ago
  The bad things are still around in your infra even if you manage to never see them because you only see the "dozens lines of code" at the front.
  - mrjin 4y ago
    Someone who really buys that "dozens lines of code" will definitely has no idea what kind of complexity it was about.
- hinkley 4y ago
  I’ve been cultivating this opinion that it’s the other way around. In any other domain, few people use crap tools to make something amazing, and I think it comes down to what you surround yourself with informs what you make. We need better tools if we want better things.
- mrjin 4y ago
  Oh sure. Can you try to set up ANY CI pipelines to make it more or less working, don't worry about "pipelines that build and test on multiple target platforms" and get back to share with us your experiences?
  - allarm 4y ago
    What exactly do you want to hear? Not op, but I’ve been doing it for some time.

Sebb767 4y ago

The big question is: Do you need that kind of functionality? I agree that very large and complex infrastructures have their place - the problem is just that they import a ton of complexity and usually cost a lot. People are always suprised when they see a minimal webserver instance on a ten year old debian [0] handling tens of thousands of requests, without issue. It might go down once a decade because the disk failed, but it cost 1200$ to run over that decade. I don't think that's the perfect way, but modern infrastructures love to include a lot of complexity when it's not needed. The problem is that including something is very easy and the cost is only payed once it breaks. Also, hardware is really cheap (you might not think so, but if you compare it to western IT salaries, it is). [0] I know outdated instances with no update strategy are not really a benchmark, but I encourage you to go out and ask people what their container base image upgrade strategy is - the situation really did not change that much.

_vertigo 4y ago
Kind of depends on what you’re building, right? Avoiding building new features in order to limit complexity is probably a good idea in select cases but not in others. Using more complex, purpose-built technology over standard boilerplate is often very cost-efficient, but only if you run at a scale where the additional complexity is worth the savings…
fipar 4y ago
> The big question is: Do you need that kind of functionality? I agree that very large and complex infrastructures have their place - the problem is just that they import a ton of complexity and usually cost a lot. I think that's the key. Most of us aren't Google/Facebook/whatever. Yet lots of people want to mimic those. A simple stack with a relation database on a single machine still goes a long way. In fact, it goes a lot longer way today than it did 20 years ago. It's ok to scale when you need to, but I think premature scaling is the wrong way to go, especially since most things never get to FB/Google/etc size.
- bluepizza 4y ago
  I understand trying to keep complexity low. But I don’t understand how a relational database would be less complex than a simple key value store. Every time this argument pops up, I end up wondering if it’s really about architectural complexity, or if it’s about sticking with the old and known.

deleted 4y ago

[deleted]

terpans 4y ago

I was working on large scale deployments with multiple datacenters, multiple databases, message passing networks and running multiple customer-facing products. Load balancers and HA setups were already in use. LVS existed. VMs were popular. "CI/CD" existed, without that name. > 2005 your average box ... Speak for yourself.

_vertigo 4y ago
- terpans 4y ago
  Indeed all of those things existed and they weren’t in widespread use because 99% of organization did not need them. And it hasn't changed. > 2 sysadmins were absolutely not managing “large scale deployments with multiple data centers” Please don't make things up because I was there. > But today it’s actually super plausible that 2 or 3 Devops with modern tooling can manage all of that. Not really.
  - _vertigo 4y ago
    - deleted 4y ago
      [deleted]
    - dijit 4y ago
      I was there too and some things were better, some thing were worse. The problem is that we’re being sold a lie and it’s hard to swallow. “Cloud will save you on staffing costs” - no, it just means you have specialists working on proprietary solutions, just like IBM mainframes or storage appliances or load-balancers/traffic-shapers of yore. “This technology will make rollouts easier” - until it breaks down and is not easy to put together again, you’re at the mercy of your upstream and you had better hope you keep a really solid update cadence and not introduce something you can’t consume later. “Layers on layers means things are composable” - sometimes, but you need more people to know more things to put it together properly. Running everything from QuickStarts is doomed to fail, but I see so much of it. Our config management tools back in the day were crummy as hell, cfengine being the only notable one (which was awful), our monitoring systems required a lot more handholding and truthfully: people were not happy to talk through requirements; which was probably why “devs run production” was appealing. these things have gotten better, but nearly everything else has definitely gotten worse.

thezilch 4y ago

Classic over-engineered DevOps, adding complexity for complexity sake. When you have a hammer... Everything you described was 2005.

HelloNurse 4y ago
In 2005 there were, for instance, fancy J2EE application servers instead of fancy container roulette tools.

rco8786 4y ago

The fun part is that the 2005 architecture is still plenty sufficient for 99% of deployments but everybody architects for 100x the scale they actually need.

tech_tuna 4y ago
This is a good point, but I'd say that's just the ops version of premature optimization.
- int_19h 4y ago
  No, it's the ops version of premature abstraction - when devs build a whole extensible framework around a simple use case just because "requirements might change later" (which 90% of the time they never do, so on average the work is wasted much more often than not).
  - tech_tuna 4y ago
    I mean yes. . . premature abstraction driven by premature optimization. Or more simply, solving a problem you don't have now, and might not ever have. Everyone talks FAANG scale on day 1. It's common and absurd.
bottled_poe 4y ago
Usually because most business say they need highly available, scalable infra from day 1.
- rco8786 4y ago
  I typically see it happen in the other direction myself. Business folks describe the problem/product and engineering team says “oh yea we can do that, we’ll need K8s” when they do not, in fact, need K8s

tech_tuna 4y ago

This times one million. Back in 2005 there would have been so much shit you wouldn't even dream of doing and a lot of things you did do, you sure as shit wouldn't do now. . . we used to write our cache systems. Migrations were all custom scripts. Fucking nightly builds were a thing because we didn't kick of builds on commit. Unit tests weren't even common practice back then. Yeah, most places had tests but there was no common language to describe them. And as much as git can be a big complex pain. . . merging was a BIG thing back then too. I seldom deal with long lived branches and the nightmarish merges they often needed. Also, to all the young folks who "want to work with physical servers" again. Have fun with the next set of device driver patches you need to roll out. I heart my containerized IaC buzzword laden cloud existence. #FuckThatNoiseOps

raffraffraff 4y ago
It's as much a cultural problem as a 2005 vs now problem because I have experience working at a joint that uses AWS yet chooses to roll their own Kubernetes, uses GitHub yet flails around with long lived branches and 20 minutes CI stages, uses containers but still has 20 minutes build times, uses ArgoCD yet has tiresome "release events" that generally can't be rolled back and take at least an hour to "fix-fowards" after a bad release. Sometimes you can lead a horse to water and even push it into the trough, but it would rather eat dust than let go of its last-decade ideas about operations.

mbesto 4y ago

> Your automation may be an order of magnitude more complex than 2005, but it enables two orders of magnitude more functionality. And can create more than 2 orders of magnitude in terms of economic value (i.e. profit) as a result.

lkrubner 4y ago
If that was true, why has the economy since 2000 never hit the speed of growth that it had in the 1990s? It seems the big economic gains of the Internet came early, and what we've seen since is the Law Of Diminishing Returns: increasing inputs for decreasing outputs. Bigger investments for less economic growth.
- mbesto 4y ago
  > hit the speed of growth that it had in the 1990s? It seems the big economic gains of the Internet came early It did, just not in the way you're describing it. This is how it's manifested itself: > Apple, Amazon, Alphabet, Microsoft and Facebook all account for just under 20 percent of the market value for the entire S&P 500. With a collective value of nearly $5 trillion, these top tech companies easily dwarf other entire industries in the index, with companies like Berkshire Hathaway and JPMorgan Chase falling well short. Currently, the total valuation of the S&P 500 is almost $27 trillion. [0] - https://www.statista.com/chart/20794/tech-companies-highly-valued-in-sp-500/ https://www.statista.com/chart/20794/tech-companies-highly-v...
  - lkrubner 4y ago
    That is not economic growth. A valuation of some companies on the stock market has noting to do with economic growth. "Create more than 2 orders of magnitude in terms of economic value" never actually happened.
    - mbesto 4y ago
      > 2 orders of magnitude in terms of economic value (i.e. profit) Then you clearly misread my initial point. economic value != overall economy growth

onion2k 4y ago

2005 Your app takes some user data, stores it in a database, and presents a filtered view of it to the user on request. 2022 Your app takes some user data, stores it in a database, and presents a filtered view of it to the user on request.

_vertigo 4y ago
Well if that’s all you’re doing in 2022, there’s nothing wrong with deploying your code and grepping logs over ssh. Bash scripts and offloading some of the complexity of management to AWS (e.g. use autoscaling) goes pretty far
- HelloNurse 4y ago
  Autoscaling? Do you mean facing so much cloud management complexity that it invites automation, in order to spend more money for a herd of excitingly ephemeral server instances than for a reasonably overprovisioned boring actual computer?
- onion2k 4y ago
  In 2002 I built a website that got about 3m page views a week, with about 50k concurrent users, and a whole bunch of fun things like payments, outbound email, etc. It ran on a single server. Since then I've working on things that use several servers, things that use no servers with edge functions instead (which have servers, but still), and that use autoscaling across lots of servers. No doubt some businesses need autoscaling but most don't. It definitely shouldn't be something you reach for before you actually anticipate a need for it.

fipar 4y ago

I partially agree with what you're saying but let's not pretend we weren't handling payments in 2005, some of us where anyway. I think what changed is the scale of things: we had a lot less people online back then. I think the increased complexity in our architectures is correlated with the DevOps role coming to the picture, but I'm not sure there's a cause link there. My recollection from having lived through the initial years of DevOps (I began working professionally in 2000) is that an important goal was to reduce the terrible friction between dev and ops (the latter including sysadmins, DBAs, etc). Whatever the extra complexity we have now, I would not want to go back to the days when I carried a pager and responded to find out the problem was caused by an application I had no insight into, and there was no corresponding on call person on the side of the dev team. Another important goal was to manage infrastructure a bit closer to how we develop software, including the big step of having the configuration (or the code that generates it) in SCM. Another thing I don't miss is logging into a server to troubleshoot something and finding /etc/my.cnf, /etc/my.cnf.<some date>, /etc/my.cnf.test, etc. It's much easier to just run tig on either the file or the ansible/chef/whatever that generates it IMHO.

democracy 4y ago

2005 was fullon enterprise websphere/weblogic era, so no, if anything it was much more complex (from architectural side) than today's python/nodejs or even spring boot solutions. Automation (bash/ant) plus ci like cruise control already there.

littlestymaar 4y ago

> you handle payments, Yeah, because obviously nobody handled any payment in 2005 … The same can be said for everything in your list. And more importantly, it's unlikely that your business is more complex than what it used to be in 2005. And if you're spending more resource to deliver the same business value, you're wasting them. (That's why PHP is still ubiquitous btw, it sucks by most standards, but it's good enough to do business and that's what matters).

Jistern 4y ago

>> Your automation may be an order of magnitude more complex than 2005, but it enables two orders of magnitude more functionality. This! The primary problems with DevOps are... 1. It is still in its infancy therefore it's changing quickly. 2. Bad (or no) documentation is much more painful than before. A single family house without blueprints can, usually, be adequately serviced; whereas, a 27 story office building cannot.

arinlen 4y ago

Also not pictured: * 2005: ship it if it builds. * 2022: run unit testing, deployment to beta, run integration testing, run UI testing, deployment to preprod, run internationalization testing, run regional/marketplace-specific testing, run consistency checks, run performance tests, deployment to prod.

takeda 4y ago
I think the variable here isn't year, but where you work(ed).
- kitd 4y ago
  Maybe, but the pipeline required to do all that in 2005 would be considerably more complex and brittle than the 2022 version using tools and infra designed for the job.

deleted 4y ago

[deleted]

dhzhzjsbevs 4y ago

Missing the point: That code in 2005 was better in every way. Context: just learning loopback API framework. What a steaming pile of overengineered garbage.

hsn915 4y ago

All you did is describe the increase in complexity of the infrastructure. Which does not rebut the point, because that is the point. The infrastructure people are using now is much more complicated than what was common 15 years ago. If you want to rebut this you need to demonstrate the kind of capabilities we have now thanks to this complexity that we could not have before. > you handle payments No you don't. Most people handle payments by integrating with a 3rd party service, most likely stripe or paypal.

LeonB 4y ago
Good point re payments. Your system today doesn’t even handle left pad, outsourcing even that to a non deterministic tree of geo-distributed modules.

wruza 4y ago

you handle payments, integrate with other services, queue tasks for later, and so on This just means “use APIs”, adds {base_url, secret_key} pairs to a config file. Where are orders of magnitude devops-wise?

barking_biscuit 4y ago
Integrating with other services makes your software more brittle as there are more points of failure from hardware and networking on your end, to API gateways, networking, and hardware on their end. Periodically those things tend to go sideways which means whatever it was your system was trying to do doesn't happen, or worse it only partially happens. Usually some poor engineer is then tasked with figuring out if this is an on-going problem or a one-off problem? Did it affect a single customer, or is it affecting all customers? Is it affecting a single instance, or is the problem happening across all instances? Is it a problem on our end, or is the 3rd party API misbehaving (again) today? Generally you need solid APM and logging to support the operations of this. Then there's queueing tasks for later and making sure you never lose messages between systems. All sorts of fun happens when you hit a capacity limit in your queueing system due to unintended/unnoticed design flaw and all of a sudden system performance starts to tank as things aren't getting processed due to too much stuff in the queue that isn't draining at a sufficient rate. You need to figure out is the queue actually draining or is it still increasing in size? Are all the worker nodes still running or did one (or several) of them hang and didn't die but simply isn't processing any jobs out of the queue but still consuming a spot in your auto-scaling group blocking a new healthy instance from coming online? Is it a problem in your software or is it actually a blip in operations from your cloud-provider? It's way, way, way more complex than simply "chuck some API keys in a file and she'll be right!". That's hobby project level stuff, not realistic ops of a production system.
- wruza 4y ago
  I’m literally working at automated finance and we are offloading these cases to client support, and late/next day status requests for b2b peers. The only devops-level technicality there is an in-process keep-alive system which prevents overwhelming the support repeatedly. What you’re describing tries to deal with these issues at the level where it is hard, and it is, but that’s exactly the road to complexity and infra costs (and infra issues as well). As a result, it trades simple client issues for hard technical ones, and you still need support. It’s cool to have an ideal incident-free system, but only when you can cover its costs - and more importantly its evolution barriers - by exactly that quality. Iow, don’t replace humans with machines either until machines learn to wipe their own ass, or until it’s just too much ass to manage efficiently anyway. I’d fight at your side few years ago, but since starting to work in this field and asking around I realized nobody cares about that ideal way, cause it’s so expensive and hard to maintain, and the cost of change is prohibitive. It even left few scars on my mental health, exactly because I was too anxious to go this way after rotating in environments where devops is praised as something inevitable to survive. If you squint at us at the right angle you still can see devops, but I think we are just expressing two opposite sets of ideas, each applicable in its own type of a business env. You may still call that a hobby-level, and I’d even agree (cause it essentially is) if that hobby didn’t bring as much as to sustain and profit the company at the levels many companies would wish they were at. If it works and brings revenue who cares how it’s categorized.

xkbarkar 4y ago

Wanted to add, todays pipelines include a whole new level of security that was pretty much ignored in 2005. Security adds complexity

terpans 4y ago
> todays pipelines include a whole new level of security that was pretty much ignored in 2005. The very opposite. > Security adds complexity If anything, complexity is the enemy of security

GrumpyNl 4y ago

The question is, do you need all that or was/is PhP and Mysql enough for the job.

HelloNurse 4y ago

Extremely complicated architectures are a liability, not a feature to be proud of; increasing complication (i.e. costs) doesn't mean increasing functionality (i.e. value). For example, why do you "queue tasks for later"? Do you have exceptionally punishing latency and throughput requirements that can only be met at a reasonable cost by not answering synchronously, or it's because your database doesn't do transactions? Similarly, what do you do with "queues, caches, and so on"? Meet further extreme performance and availability requirements, or attempt to mask with additional complications the poor performance of inefficient complicated components? In 2005, but also in 2000, web applications had already progressed past "just serving a web page", mostly without complicated architectures and therefore without the accompanying tools. I think tool improvements made cargo-culting actually advanced state of the art software architectures and processes easy and affordable, creating a long term feedback loop between unnecessary demand for complexity (amateurs dreaming of "scaling up") and unnecessary development and marketing of advanced (but not necessarily good) complexity-management tools, often by the same amateurs.

tsarchitect 4y ago

Also not pictured-- 2005: two system administrators wrote all automation using a handful of Bash, Perl and Python scripts AND LEAVE THE COMPANY a couple of years laters 20xx: new hired system administrators continue to rewrite scrips. no shared knowledge because scortched-earth policy in effect 2022: HN: DevOps is a failure -- you just need two system administrators...

phendrenad2 4y ago

> you handle payments, integrate with other services, queue tasks for later I distinctly remember buying things on Amazon in 2005. Actually come to think of it, I did everything you list in 2005, at multiple companies.

bayindirh 4y ago

I find this take apologetic. "But we're running on the cloud, we need this complexity", is a wrong take. Yes, the underlying services may need this, but the infrastructure managing these services don't need to be this complex. Every complex tool can be configured to work in a simpler manner, and all configuration repositories directing these tools can be organized much better. It's akin to a codebase after all. Many code quality processes and best practices to apply to these, but sysadmins don't like to think like developers, and things get hairy (I'm a sysadmin who does development, I see both sides of the fence). The sad part is, these allegedly more robust platforms do not provide the transparency neither to developers, nor to sysadmins, nor to users. In the name of faster development, logging/debugging gets awry, management becomes more complex and users can't find the features they have used to have. Why? We decoupled systems and decided to add this most used feature at a later date, because it's not something making money. Now my account page even doesn't show my e-mail address, I can't change my subscription renewal date, or see my past purchases, why? They're at distant tables or these features are managed by other picoservices which are not worthy to extend or add more communication channels. Result? Allegedly mature web applications with weird latencies and awkwardly barren pages, with no user preferences or details. Adding layers and layers of complexity to patch/hide shortcomings of your architecture doesn't warrant a more complex toolset to manage it.