4 ms·
A Jenga tower about to collapse: Software erosion is happening all around us
- EdwardDiego 2y agoA very vague article about how our entire digging infrastructure is about to collapse, from someone looking to sell you a spade.
- louwrentius 2y agoReads like a chat-gpt prompt
- dredmorbius 2y ago<https://news.ycombinator.com/item?id=23486170 https://news.ycombinator.com/item?id=23486170>
- keyle 2y ago- And tonight at 11... - Dooooooom!
- ungamedplayer 2y agoOnce again we blame developers for corporations not testing deployments and pushing changes live. Nobody can test everything. Big deployments require big testing.
- lewdev 2y agoThis article is capitalizing on the Crowdstrike incident. It was costly but a mistake. As a software engineer, I just know that's all it is. I don't think there is a upward trend of these mistakes because they are always trying to be careful and sometimes they also get careless. Some additional processes might be added to avoid it, but years later it may happen again somewhere else in another company. I don't think it's because of "software erosion." And the recovery was a costly day or two but it was fixed and we all went back to normal.
- lnenad 2y agoI think we can look at the issue from two sides. There is definite enshittification of software happening all around us with companies unable to understand that an end goal of product development could be achieved and focusing on feature bloat to protect them from up-and-coming startups taking a piece of their cake. This means that both good features and bad ones get added and things have to change constantly making the entire end user experience worse. This complicates things on the softdev side as well as tech debt grows, architecture was made without taking into account some of these features, QA is harder to do well considering the larger surface area. So this leads us to this dystopian view of how things are and when a mistake happens an echo chamber could be easily formed that makes these views (software sucks) feel like postulates. On the other hand we've never been surrounded by so much software in history and it keeps growing, and will keep growing and so far the earth is not collapsing. There's so much that depends on people typing code into their editors it's truly amazing we've reached this point. Keeping everything afloat in this new reality is increasingly difficult as many of these systems work together and require a broad understanding of many domains (not every product/company has budgets for multiple roles so you have one person that does infra/code/qa with a multitude of tools) to enable them to work without issues. So the number of interactions people have with code is increasing and therefore when problems with software that is used by a lot of customers come up it becomes *very* visible and feels like nothing is working. But in reality several thousands of microprocessors in very close proximity of these people keep chugging along and their phones, payment cards, headphones, monitors, tvs, speakers, smart *x*s, coffee makers, thermostats etc... are as reliable as they ever were with a lot more to offer so the other view could also be very realistic (software has never had this level of quality).
- lesuorac 2y agoKeep in mind, an "additional processes" that would've avoided the Crowdstrike incident was a smoke test on a Windows machine. Literally if you just checked if the binary + config didn't break a windows machine before pushing to 100%. There is definitely software erosion. Stores didn't sell CDs of software that just literally crashed your machine when you installed it.
- 2y ago
- MrThoughtful 2y agothe average developer spends 42% of their work week on maintenance Indeed, I see that happening all around me when I watch how my friends build their startups. The first few months they are productive, and then they sink deeper and deeper into the quicksand of catching up with changes in their stack. So far, I have done a somewhat good job of avoiding that. And I have a keen eye on avoiding it for the future. I think a good stack to use is: OS: Debian DB: SQLite Webserver: Apache Backend Language: Python Backend Library: Django Frontend: HTML + CSS + Javascript Frontend library: Handlebars And no other dependencies. I called Django a "library" instead of a framework, because I do not create projects via "django-admin startproject myproject" but rather just do "import django" in the files that make use of Django functionality. In the frontend, the only thing I use a library for is templating content that is dynamically updated on the client. Handlebars does it in a sane way. This way, I expect that my stack is stable enough for me to keep all projects functional for decades to come. With less than a week of maintenance per year.
- ChrisMarshallNY 2y agoI probably spend well over 50% of my time testing for (and finding) issues. In my experience, finding issues is 99% of the time involved in fixing them. Once I find the cause of a problem, the fix is almost immediate. > Thank the dependency hell we’ve put ourselves in. This was something that we knew was coming, ten years ago. Modular programming is an old, well-established discipline, but that generally means modules of code that we write, or that we control. I write most of my stuff, using a modular architecture, and I write almost every single one of my reusable modules. Things are very, very different, when we import stuff from outside our sphere. Heck, even system modules can be a problem. Remember "DLL Hell"? When I first started seeing people excitedly talking about great frameworks and libraries they use, on development boards, I was, like "I don't think this will end well." I just hope that we don't throw out the modular programming baby with the dependency hell bathwater.
- bilekas 2y ago> I think a good stack to use is: These things are relative though, also futureproofing your stack will run into issues later if you need to scale to N number of requests etc. When you are thinking forward for optimizations, you may re-evaluate your thoughts on the right stack.
- irjustin 2y agoThe article actually discredits its own conclusion early on: > These outages didn’t happen because developers didn’t test software. The conclusion being: > How do you get quality code?...Don’t skimp on static code analysis and functional tests, which should be run as new code is written. But even working from the conclusion backwards, which is "specs+code analysis" will save you from the big scary thing of "software erosion" and "complexity" thusly sparing us all from outages, I disagree. Specs+analysis are helpful, but they do not magically solve complexity at scale. Crowdstrike sure, would've benefited from testing I agree but so many other large outtages need more than that, which is the disconnect of the article for me. At some point you need blackbox, chaos monkey level production tests. Bring down your central database, bring down us-east-1. What happens to the business? I'm not sure if this is valid, but a lot of the savvier tech companies' outtages feel like they're router configurations that lead to cascading traffic issues. But I have no data to back this thought up.
- josefrichter 2y ago"average developer spends 42% of their work week on maintenance" – is that true (source)? Is that your personal experience too?
- lonelyasacloud 2y agoThe figure will vary somewhat depending on things like dev experience, previous coding practices, automated testing etc, but there is a corpus of research that suggests that a ballpark figure is somwhere around that 40% is about right. Here's what Perplexity says about it https://www.perplexity.ai/search/what-evidence-is-there-that-th-Dwvg_iSqToayHSmlWwNjtA https://www.perplexity.ai/search/what-evidence-is-there-that.... Seems plausible here, particularly if allow for the often considerable additional preemptive Maintenance effort required trying to avoid breaking the existing product while adding new features.
- poikroequ 2y agoThe bigger problem I've seen is high turnover rates in the industry. The people who built and know the system leave. There wasn't a sufficient window for KT (knowledge transfer), so you're left with a bunch of new devs who only have a surface level understanding of the code and architecture. Productivity drops severely because every new feature requires several hours of reading code / reverse engineering. Then these new features often break other things because the devs don't know the intricacies of the system, so many more hours are spent fixing the bugs.
- ChrisMarshallNY 2y agoI agree. The distressingly short tenure of staff, these days, causes many problems. The solution is not so easy. There's a real reason that people have so little loyalty to employers. I feel that the first move needs to be made by employers. They need to give people a reason to stay (and it is not always money, but that is a big motivator). When they do that, some employees will take advantage of their employers, and that needs to be factored in, at the beginning. None of this "Lazy Bob is the reason I'm screwing you all." stuff. I believe that collective punishment is a war crime, but we do it all the time, in business. We need to come up with smart, adaptable, heuristic policies that work for all employees; not simplistic "One size fits all" HR policies that make lawyers happy. And employers need to stop deliberately screwing their employees. Eventually, things will settle out, where that's rewarded, but it might take a long time to get there. But that's just a dream. I am under no illusion that it will actually happen. Instead, we can look forward to decades of Jurassic-scale disasters, and after-the-fact hand-wringing.
- whaleofatw2022 2y agoI see a related issue, but Moreno with contractors. Companies outsourcing new dev to random houses and expecting the in house people to fix bugs when they weren't even on the pull requests half the time.
- from-nibly 2y agoSee also programming as theory building.
- SebFender 2y agothis has been the situation in computing in general for many decades. When I started in the 90's - maintaining different unix distributions was a continuous package game. Now looking at ops and devs at companies, this has been the ongoing work. I think it's just an integral part of computing and one of its core challenges...
- kennu 2y agoIn my view, this is why cloud exists. You can externalize as much of your software stack as possible to the cloud platform, and only implement and maintain yourself the parts that differentiate. On AWS, this means using Lambda, Step Functions, AppSync, API Gateway, DynamoDB etc. and letting the cloud provider worry about maintaining most of the technology stack.
- blueflow 2y agoImplied in this: That the cloud provider can do it better than you. Not universally true nowadays.
- fergie 2y agoSome of us remember The Great AWS Outage of 2010.
- kennu 2y agoBut we tend to forget all the outages we had while self-hosting datacenters and servers before 2010 ;-) Of course, times also changed since early days, so that it is more and more important to be up 24/7.
- ornornor 2y agoWas it also OVH that lost their DC and all backups in a fire a few years ago? They took entire businesses down with them.
- gwynforthewyn 2y agoI’m actually just cynical enough this morning to think this was written by a chat AI that was prompted with “write a few paragraphs of ragebait about software erosion”. The article asserts software erosion is happening for a series of vague reasons like “dependency hell” and that developers are asked to add new features to codebases. Developers have been adding features to codebases for _decades_. It’s a demonstrably fine activity. The article doesn’t chain together “practice X causes bad effect Y”, it just says themed sentences one after the other that don’t follow a reasoned argument. There aren’t even any personal anecdotes. There’s so many people writing much better instructive content; it’s a little heartbreaking seeing nonsense like this elevated.
- mistermann 2y ago> Developers have been adding features to codebases for _decades_. It’s a demonstrably fine activity. See also science, industrialization, and the temperature of the planet.
- fuzzfactor 2y agoBefore computers proliferated, it was supposed to be a promising future for one reason because it would be theoretically possible to write a single program which truly solved a problem (big or small), and that problem would remain solved forever. I guess you would have to qualify that, as long as no undesirable changes in underlying hardware or OS were to be contemplated. But why would you want that anyway if you were interested in long-term solutions? Plus if all you needed to do was one thing (big or small) with a PC, you would be fine with a PC that was only capable of running one program at a time. Think about the user experience back when you were anticipating PC's to one day become ubiquitous. Ideally you would be able to go up to any PC anywhere (or anywhere you have permission in secure environments) with your portable storage device, floppy, USB, whatever, and just run your executable to accomplish what you want without any other dependencies from a network or anywhere else. The more technical the user the more they realized this meant that a program should be truly finished before deployment, not subject to revision or updates, have no dependencies, and in no need of further maintenance. Why settle for less? Regardless of whether or not it could ever be done like that any more, I would still want my software to come from people who would be outstanding at this approach anyway. At the other end of the spectrum you have the "update enthusiasts". It's not bad as long as there are never any "breaking changes", only new features, but that is far too rare. What bothers me is deploying half-baked stuff to begin with, and then needing years of updates just because that's such a well established trend. Looks to me like complete superstition, and the foundation is becoming less firm all the time as a result. >See also science, industrialization, and the temperature of the planet. Good examples for better or worse. Sometimes "breaking changes" get out of hand and scale themselves, which is what people just aren't avoiding well enough any more in many ways. It takes concentrated effort not to do it.
- mysal 2y agoIn the current climate of cultural revolution experts are forced to be silent, hand over their authority to mediocre politicians and let everyone commit for the sake of "equity" (meaning: the politicians have an income). No wonder that the whole system collapses. To be fair, in the 1990s software wasn't great either, but many things were new and written under enormous time pressure like the Netscape browser. Linux distributions were best around 2010. Google and Windows were best around that time, too.
- BriggyDwiggs42 2y agoIn what way are experts made to be silent on any practical matter?
- ahdqk 2y agoExperts are allowed to silently fix mistakes of the politicians. They are not allowed to criticize them for bad practices, even in the case of major mistakes. The coercive measures are vicious verbal backlash from the politicians, expulsion from a project, not being promoted or being fired as a "problematic" person.
- BriggyDwiggs42 2y agoWhat are some specific examples? I see experts criticize politicians all the time.
- LordShredda 2y agoJust because firefox sucks doesn't mean the global economy is collapsing
- snapcaster 2y agoIs this an actual contributing factor? I find it hard to believe some vague political ideology popular in wider society is a major issue when it comes to software development
- 2y ago
- TacticalCoder 2y ago> the average developer spends 42% of their work week on maintenance Don't worry: AI will do coding maintenance / bug fixes soon.
- kkfx 2y agoThe root problem is simply the modern, old, OS++ concept, or the divide et impera commercial concept. Classic systems was a single application, fully integrated. This design create a slow, incremental evolution and a plethora of small improvements, the commercial design of compartmentalized levels have created a Babel tower of dysfunctional crap mostly trying to punch holes between levels. An example: a single NixOS home server can do on a very small system what a modern deploy with docker and co can do on an equivalent starship, a simple Emacs buffer, let's say a mail compose one, can allow quickly solving an ODE via Maxima, while a modern Office Suite can't without manual cut and paste and gazillion SLoC more, a Plan 9 mail system do not need to implement complex storage handling and network protocols, all it need is just in the system, mounting someone else remote mailbox, save a file there it's sending a message, reading a file from a mounted filesystem is reading one and the same is for viewing a website. In Gnus a mail, an RSS article and NNTP post are the same, because they are DAMN THE SAME, a damn text with optional extras consisting of a title and a body. That's the power of simplicity we lost to push walled gardens and keep users locked-in and powerless. The modern commercial IT is simply untenable: - even Alphabet can't scan the whole web, a distributed YaCy on gazzilion of homeservers can though, and with MUCH LESS iron and costs for the whole humanity; - nobody can map as in an OSM model, since anybody doing it share everything on every other; This is the power of FLOSS. It's time to admit that we can't afford a commerce/finance managed nervous system of our societies, simply.
- eliasson 2y agoI am not a fan of comparing bad software with erosion or organic decay. It's feels like avoiding responsibility. Software is software, it is made worse by people, nothing else.
- ornornor 2y agoMitigating and coming up with a plan to remedy these issues was my specialty over the 15ish years I wrote software professionally. All these initiatives and plan always does when it reaches an executive reacting with « it works now why should we spend any money on not making new features? We’re not doing your gold plating we don’t need it » I eventually got tired of this, ran out of motivation, and quit software engineering. MBAs who understand nothing about software treat software developers as code monkeys and then we are in this situation. I’m still bitter about the whole thing and how it completely put me off writing software (which I used to love doing). Some days, I’m cheering for these failures and crashes imagining some exec somewhere will eat a big shit sandwich for causing it. But I’m not kidding myself, I know it’s the software engineers getting blamed and working over time for these outages…
- sumuyuda 2y ago> These outages didn’t happen because developers didn’t test software. Funny how there is no mention of how modern tech companies offshored/outsourced and even fired manual QA testers. Developers aren’t testers. Do we expect a civil engineer to test the bridge they created before opening it to the public? Also, with a move fast and break things mentality, stable and quality software went out the window for a continuous release of broken/buggy software.