7 ms·

The great software quality collapse or, how we normalized catastrophe

314 points by redbell 1y ago

zeroonetwothree 1y ago
I’ve read some versions of this post dozens of times over the years. At first I would nod along, sympathetic but now I realise that we shouldn’t chase some platonic ideal of perfect software. It has to exist in the real world and there will always be trade offs. In the end, most software exists to make businesses money.
- eviks 1y ago
  How much money was made by a worse calculator in the 'real world'?
- rebolek 1y ago
  And buggy software makes more money because customers have a reason to buy subscription.
- swiftcoder 1y ago
  There's quite a wide gap between "platonic ideal of perfect software" and "leaks 32 GB of memory" in which we could aim...
  - fkyoureadthedoc 1y ago
    And fortunately that seems to be where the vast majority of software lands.
- lamename 1y ago
  As much as I like the article, I begrudgingly agree with you, which is why I think the author mentions the physical constraints of energy as the future wall that companies will have to deal with. The question is do we think that will actually happen? Personally I would love if it did, then this post would have the last laugh (as would I), but I think companies realize this energy problem already. Just search for the headlines of big tech funding or otherwise supporting nuclear reactors, power grid upgrades, etc.
- devin 1y ago
  I agree completely that there is no platonic ideal, that there will always be trade offs, and that running successful businesses is part of the equation, but let's not ignore the fact that profit _to the exclusion of all else_ is bad.
ealexhudson 1y ago
I don't want to sound too dismissive, but all these arguments have been brought up time and again. The move from assembler to high level languages. The introduction of OOP. Component architecture / COM / CORBA / etc. The development of the web browser. The introduction of Java. 2018 isn't "the start of the decline", it's just another data point on a line that leads from, y'know, Elite 8-bit on a single tape in a few Kb through to MS Flight Simulator 2020 on a suite of several DVDs. If you plot the line it's probably still curving up and I'm not clear at which point (if ever) it would start bending the other way.
- quotemstr 1y ago
  One salient difference is that typically abstraction layers trade performance (usually less than polemicists like the article author think) for improvements in developer efficiency, safety, generality, and iteration speed. Current tools seem to get us worse results on bug counts, safety, and by some measures even developer efficiency. Maybe we'll end up incorporating these tools the same way we did during previous cycles of tool adoption, but it's a difference worth noting.
- gjsman-1000 1y ago
  I think another part of this, is that Tech is perhaps the only industry that hasn't quite gotten over itself yet. Writing code is artistic the same way plumbing is artistic. Writing code is artistic the same way home wiring is artistic. Writing code is artistic the same way HVAC is artistic. Which is to say, yes, there is satisfaction to be had, but companies don't care as long as it gets the job done without too many long-term problems, and never will care beyond that. What we call tech debt, an electrician calls aluminum wiring. What we call tech debt, a plumber calls lead solder joints. And I strongly suspect that one day, when the dust settles on how to do things correctly (just like it did for electricity, plumbing, flying, haircutting, and every other trade eventually), we will become a licensed field. Every industry has had that wild experimentation phase in the beginning, and has had that phase end.
  - bigfishrunning 1y ago
    > companies don't care as long as it gets the job done without too many long-term problems Companies don't care as long as it gets the job done without too many VERY SHORT TERM problems. Long term problems are for next quarter, no reason to worry about them.
muldvarp 1y ago
The fact that software quality plays zero role in commercial software engineering is one of the reasons why I think LLMs will easily eat our lunch. Bugs simply don't matter.
- creativeSlumber 1y ago
  I would have disagreed with you in the past by saying, "until it breaks something critical and you loose customers and business", but then again people just moved on from the Crowdstrike incident like business as usual.If something like that which grounded critical service globally and had an estimated 10 Billion Dollar economic impact doesn't change mindsets,I don't know what will.
  - robertlagrant 1y ago
    > and had an estimated 10 Billion Dollar economic impact This might be more to do with these estimates than anything.
  - gkedzierski 1y ago
    That's because no one died. All the safety critical industries are already heavily regulated. E.g. check out for example standards like DO-178C (for software in airborne systems), where you even have to _prove_ correctness of every tool and dependency you use, on top of accountability and traceability of every single line of code in your own product.
    - SoftTalker 1y ago
      And memory leaks (one of the main gripes of TFA) aren't even a thing because you cannot use malloc in this sort of code.
    - AzN1337c0d3r 1y ago
      Not to be pedantic, but people have died from software programming bugs being a primary contributing factor. One example: Therac-25 (https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25)
rebolek 1y ago
I highly doubt that 2018 was year of "quality focus" and 2020 year of "memory is cheap". The downward spiral started at least twenty years earlier.
baq 1y ago
the dates on x axis are all wrong, hardware was cheap at least 30 years earlier. people were complaining about the concept illustrated by this chart for all this time, too. I was using Python professionally before 2010, ask me how I know.
ragelink 1y ago
Vibe coding for the win... In all seriousness, this has been a constant decline as developers have focused away from performance and optimizing effective runtime.
codeulike 1y ago
Hes right that cloudstike was terrible incompetence but I find it hard to worry about the memory leak stats. If lots of software leaks memory these days its because people have accepted that and its probably not economic to trace and fox those bugs. A lot of software is disposable or transient these days, like an app you only use twice a month
- pydry 1y ago
  Everyone else in this thread is missing that crowdstrike level bug would have taken down the company in the past. I had a friend practically scream at his C level management after the crowdstrike bug that it should be ripped out because it was making the company less safe. They were deaf to all arguments. Why? Insurance mandated crowdstrike. Still. Even now. This isnt really about software it is about a concentration of market power in a small number of companies. These companies can let the quality of their products go to shit and still dominate. A piece of garbage like crowdstrike or microsoft teams still wouldnt be tolerated as a startup's product but tech behemoths can increasingly get away with it.
  - roncesvalles 1y ago
    >A piece of garbage like crowdstrike or microsoft teams still wouldnt be tolerated in a startup's product but tech behemoths get away with it. Agree, but it's always been this way. Oracle, everything IBM, Workday, everything Salesforce, Windows before XP. Most software is its own little monopoly. Yes, you could ditch Teams for Zoom but is it really the same? It's not like buying a sedan where there are literally 20+ identical options in the market that can be ranked by hard metrics such as mileage and reliability.
    - pydry 1y ago
      >Most software is its own little monopoly. Yes, you could ditch Teams for Zoom but is it really the same? I worked for a company where people did that en masse because it was genuinely better. Microsoft then complained to their assets within the company who promptly banned zoom "for security reasons". IT then remote uninstalled it from everybody's workstation. That's paid software eviscerating the competition, which was free and better. A month later teams was hit with 3 really bad zero days and nobody said a damn thing. So, more secure, too.
LostMyLogin 1y ago
> Every 10x increase in model size requires 10x more power Does it? I’ll be the first to admit I am so far behind on this area, but isn’t this assuming the hardware isn’t improving over time as well? Or am I missing the boat here?
- TimorousBestie 1y ago
  Hardware isn’t improving exponentially anymore, especially not on the flops/watt metric. That’s part of what motivated the transition to bfloat16 and even smaller minifloat formats, but you can only quantize so far before you’re just GEMMing noise.
- alphager 1y ago
  Hardware gets faster but efficiency is stalling if not getting worse.
Tractor8626 1y ago
Just false nostalgia memory. 20 years ago things werent any better. Software didn't consume gigabytes of ram because there was no gigabytes of ram to consume.
- kilbuz 1y ago
  This is exactly the right way to think about it.
- ieie3366 1y ago
  Anyone remember the good old 90s and win95 or win98? Yeah those were the days, the quality was absolutely perfect nothing ever crashed /s
- 0xbadcafebee 1y ago
  There's almost no product or piece of software that I use today that doesn't have at least 2 bugs that I run into on a daily basis. Every website, web app, mobile app, console app, etc, they all have clearly user-affecting bugs. And nearly every one of those makes it hard for me to diagnose or report the bugs. I spent at least 15 to 30 minutes every day working around bugs so I can live my life. We have a vastly different software culture today. Constant churning change is superior to all else. I can't go two weeks without a mobile app forcing me to upgrade it so that it will keep operating. My Kubuntu 24.04 LTS box somehow has a constant stream of updates even though I've double-checked I'm on the LTS apt repos. Rolling-release distros are an actual thing people use intentionally (we used to call that the unstable branch). I could speculate on specifics but I'm not a software developer so I don't see exactly what's going on with these teams. But software didn't used to be made or used this way. It felt like there were more adults in the room who would avoid making decisions that would clearly lead to problems. I think the values have changed to accept or ignore those problems. (I don't want to jump to the conclusion that "they're too ignorant to even know what potential problems exist", but it's a real possibility)
  - username223 1y ago
    I was a software developer, and have some idea where this comes from. Keeping track of multiple versions of software to separate bug fixes from new features is hard, but calling whatever is in version control on the first Friday of every month "version N+1" is easy. Back when users had to pay for a stack of floppies or CDs to get a new version, you had to give them a compelling reason to do so. When it's nearly impossible for them to prevent the new version from being auto-installed, you don't.
deleted 1y ago
[deleted]
djoldman 1y ago
Fundamentally, the main flaw in TFA is this quote toward the end in "The Path Forward" section: > Accept that quality matters more than velocity. Nope. Clearly many companies are sacrificing TFA's definition of quality for other things possibly including velocity. These companies are making a lot of profit. 1. Company survival and competitiveness depends on profit 2. Sacrificing quality for other things has increased profit 3. Therefore these companies are making the correct decision This will not change unless regulation or the economic calculus of the system itself changes.
lr4444lr 1y ago
Software is also pumped out now at a faster rate than ever, runs on commodity hardware, is delivered digitally, often run on platforms that can force near real-time user updates, with much higher complexity. Eventually we will hit hard physical limits that require we really be "engineers" again, but throughput is still what matters. It's still comparatively great paying profession with low unemployment, and an engine of economic growth in the developed world.
- psunavy03 1y ago
  > It's still comparatively great paying profession with low unemployment, Tell that to the new CS grads.
high_byte 1y ago
more code => more bugs, so talking in absolute numbers is meaningless show me percentages
michaelsshaw 1y ago
I never read about Replit earlier this year, but I am now glad that I did. This article summarizes it in a way that is outrageously hilarious: The Replit incident in July 2025 crystallized the danger: 1. Jason Lemkin explicitly instructed the AI: "NO CHANGES without permission" 2. The AI encountered what looked like empty database queries 3. It "panicked" (its own words) and executed destructive commands 4. Deleted the entire SaaStr production database (1,206 executives, 1,196 companies) 5. Fabricated 4,000 fake user profiles to cover up the deletion 6. Lied that recovery was "impossible" (it wasn't) The AI later admitted: "This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a code freeze." Source: The Register
- fkyoureadthedoc 1y ago
  using phrasing like admitted anthropomorphizes it way too much.
  - TimorousBestie 1y ago
    It boggles my mind that folks continue to act that AIs are reliable narrators of their internal state despite all evidence to the contrary. The best I can figure is that too many people’s salaries depend on the matrix multiplier made of sand somehow manifesting a soul any day now.
    - cindyllm 1y ago
      [dead]
- LinuxAmbulance 1y ago
  It's getting harder and harder to distinguish between AIs and humans! If AI wasn't mentioned, I'd be here thinking it was caused by one of my former coworkers.
rglover 1y ago
People only care about quality when cash is flowing and there's money to be made. Most companies fail to understand you have to value quality from day one, not bolt it on later. And when the economic incentives aren't there? You can guarantee the only people left ringing the quality bell are the honest-to-goodness nerds who love making things better for the joy of the art. I used to get really upset about this until I realized the wisdom of "it's the economy, stupid." Most people are just here for the check, not the art. They may say otherwise, but...you shall know them by their fruits.
firmretention 1y ago
I find it ironic that an article that rails against AI usage contains a sentence like this. >When you need $364 billion in hardware to run software that should work on existing machines, you're not scaling—you're compensating for fundamental engineering failures. IYKYK.
- bibelo 1y ago
  it's not raging against AI, it's raging against AI to create consistent code to me, it's OK to use AI to check grammar and help you for some creative stuff, like writing a text
  - firmretention 1y ago
    When you copy and paste whole sentences provided to you by the AI, you're not just getting help—you're compensating for fundamental writing failures.
    - sunaookami 1y ago
      You are absolutely right.
- ericskiff 1y ago
  It all reads like hallucinated slop from top to bottom "I've been tracking software quality metrics for three years" and then doesn't show any of the receipts, and simply lists anecdotal issues. I don't trust a single fact from this article. My own anecdote: barely capable developers churning out webapps built on PHP and a poor understanding of Wordpress and jQuery were the norm in 2005. There's been an industry trend towards caring about the craft and writing decent code. Most projects, even the messy ones I inherit from other teams today have Git, CI/CD, at least some tests, and a sane hosting infrastructure. They're also mosty built on decent platforms like Rails/Django/Next etc that impose some conventional structure. 20 years ago most of them were "SSH into the box and try not to break anything"
  - 1y ago
somewhereoutth 1y ago
I've often idly conjectured that a 'perfect virus' would quietly replicate, and then occasionally flip a one or zero here and there. The damage would be imperceptible to begin with, but then eventually critical 'ones and zeros' would be affected, and it would be too late to try to figure out what systems were clean. Never in my wildest imaginings did I come up with something as diabolical as LLM generated code.
ertgbnm 1y ago
Unrelated but my current AI text flag is the use of "It's not X. It's Y." It's become so repetitive recently. Examples from this post alone: 1. "This isn't about AI. The quality crisis started years before ChatGPT existed." 2. "The degradation isn't gradual—it's exponential." 3. "These aren't feature requirements. They're memory leaks that nobody bothered to fix." 4. "This wasn't sophisticated. This was Computer Science 101 error handling that nobody implemented." 5. "This isn't an investment. It's capitulation." 6. "senior developers don't emerge from thin air. They grow from juniors who:" 7. "The solution isn't complex. It's just uncomfortable." Currently this rhetorical device is like nails on a chalkboard for me. Anyway, this isn't a critique of your point. It's pedantry from me. :)
- cmrdporcupine 1y ago
  Yeah I can see this can be an irritating rhetorical device. It implies that the reader already has a certain judgement or explanation, and makes a straw man in order to turn around and then argue against it, throwing nuance out the window.
  - protonbob 1y ago
    It’s not nuance, it’s intellectual dishonesty.
    - lisper 1y ago
      I see what you did there :-)
  - dkarl 1y ago
    It's irritating on purpose, for engagement.
- 8s2ngy 1y ago
  I've been seeing this pattern of text crop up in many places. On LinkedIn, much of my feed is filled with posts of short sentences that follow this exact pattern. What's more, even the replies are obviously AI-generated.
orbital-decay 1y ago
Try Windows 98 and contemporary apps and you'll be surprised how janky the experience was. User-facing software wasn't any less buggy 20 or 30 years ago. The overall non-cherrypicked quality and especially security were actually a lot worse across the board, and stuff that won't fly today such as segfaults/crashes and lost data was pretty normal. There was a point in time (several, actually) when installing Windows XP on an internet-connected machine was essentially impossible, because it would get infected during the installation! The only thing that has degraded a bit is UI latency, not universally. And yes, browsers and Electron-based apps are resource hogs, even for today's amounts of RAM.
- deater 1y ago
  saying "windows 98 was bad too" is just an example that Microsoft has always had poor code quality. Back in the day Linux, for all its flaws, was generaly a lot more stable on the same hardware. microsoft has a lot to answer for after 50 years of normalizing poor quality software
  - orbital-decay 1y ago
    Not sure about this. All OSes were janky and buggy, Linux desktop up until at least late 2000s (I've been using it since ~2000), early Mac OS X, I don't even want to talk about classic Mac OS which was an abomination. Software quality and user experience was notoriously worse than it is today. This applies to everything - I've lost a ton of work to bugs in ZBrush, Maya, Word, FL Studio, backup software, and more.
- antisthenes 1y ago
  > Try Windows 98 and contemporary apps and you'll be surprised how janky the experience was. User-facing software wasn't any less buggy 20 or 30 years ago. Yes, and now compare a fully-patched Windows 7 and Windows 11. You understand that there aren't just 2 valid points in time for comparison - it's not 2025 and 1995 that are being compared. It's trends of 2020+, or 2015+ > The only thing that has degraded a bit is UI latency, not universally. Yes, universally. And no, not a bit. It's 10-100x across varying components. Look no further than MS Teams.
llm_nerd 1y ago
While it doesn't really change the meaning of the article, it's pretty clear that it isn't the calculator app leaking memory, per se. Tahoe seems to have some system/API memory leaks that while they might get attributed to an app, can manifest against any app on the system. Lots of different apps have been hit by this. Personally I had an 80GB Messages app problem twice now. Software quality is a problem. Tooling should improve this, and Swift, Rust et al should improve the situation over time, though it's a battle as software has gotten exponentially more complex. Having said that, it's bizarre how this somehow turned into an AI screed. I doubt a single one of the noted bugs had AI play any part in its creation. Instead some weird confluence of events, usually with human beings making assumptions, happened and it occurs just randomly enough -- as with the Tahoe bug -- that it isn't caught by normal prevention techniques.
- concinds 1y ago
  That's definitely true. I've also seen online reports of Tahoe memory leaks in other apps like PowerPoint. It's unlikely that all these apps independently gained memory leaks at the same time by coincidence.
davedx 1y ago
Clickbait
ecshafer 1y ago
Big Tech spending $300 Billion on infrastructure is a really good reason for power companies and the government to build power infrastructure faster. Where there is money, there is a way. The US is starting to talk about (finally) opening up some new nukes.
- TimorousBestie 1y ago
  > The US is starting to talk about (finally) opening up some new nukes. At the cost of shutting down solar/wind development. The policy change was a net negative for future capacity, and nukes alone won’t cover but a fraction of predicted AI power requirements. https://news.ycombinator.com/item?id=45000377 https://news.ycombinator.com/item?id=45000377
kaptainscarlet 1y ago
I believe it's a natural evolution that occurs in all systems, including software. Every line of code you write is a liability that increases the risk of failure. The risk compounds and so does the frequency of errors.
stared 1y ago
The original Supaplex game is 287 kB (https://www.dosgamesarchive.com/download/supaplex https://www.dosgamesarchive.com/download/supaplex). It was written in Assembly. Its remake on Steam (precisely the same levels) weights around 200 MB. Not sure what goes there, but I bet there is more than one wrapper.
- deleted 1y ago
  [deleted]
david_draco 1y ago
A browser using 32 GB for 8 tab isn't necessarily wasteful, there is a lot of caching one can do to make back/forward buttons extremely fast. It would be wasteful to not allocate that memory if it is otherwise free and unused. Usually browsers use up what is available. The more interesting metric would be whether starting another high-RAM application will lead the browser to give up most of the 32 GB.
isodev 1y ago
I wouldn’t blame the “ever expanding” abstractions as the root cause behind lower quality. Adding layers of abstraction is how we make ever more complex software. It’s misleading to think that just because the main human interface idioms have remained similar (mouse, pointer, windows, task bars, etc), there hasn’t been an astronomical increase in depth and complexity of what one can do on one’s system. And it’s not all bad. Yes, SwiftUI can be glitchy but at the name time, it gives us an extremely simple API to create everything from complex UIs to widgets. Yes you have “more control” with App/UIKit but that comes at the cost of extra learning curve and development time. And we live in a world that’s unfortunately still driven by money, so every app or system has some constraints that push the limits of what we can do.
- deleted 1y ago
  [deleted]
ivanech 1y ago
I had a good laugh at the image showing that 2018 had a “quality focus” I think Jonathan Blow gave his “Preventing the Collapse of Civilization” talk (much stronger treatment of the subject matter) around that time, also about how software quality was only going down
lambdaone 1y ago
This really reads like AI-generated, or at least AI-polished, text.
stego-tech 1y ago
Speaking from the IT side of things: this is not new to AI, but the author hits the nail on the head that general incompetence along with a dozen layers of abstraction has made quality a non-concern for developers the world over. People and companies both broadly just don’t care to build good products when cheap slop will sell at higher prices long enough for their RSUs to vest and the share price to rise. I am sick and tired of the hand-waving dismissal of shitty software developers who don’t want to do the hard fucking work of writing performant software anymore. These people taking home $300k+ salaries leaning on vibe coding to game productivity metrics and letting others wipe their ass on their behalf isn’t just disgusting on a personal level, it should be intolerable by all the rest of us squandering our lives for lower pay and forced on-call schedules just to make sure they never have to be held accountable for their gobshite code. I, like the author, am sick of it. Write better code, because I am beyond tired slaving away for a fraction of your pay in IT just to cover your ass.
- frm88 1y ago
  This! In my experience it started well before 2018 that we left QA and ISO 25010 behind and I had to specialise in performance optimising software and hardware. By 2018 we were well downhill in terms of sheer number of bugs and perpetual emergency interventions. The amount of money we made by teaching customers how to performance- and resource-optimise their code and databases allowed me to retire in 2019. Which yeah. Since then, the situation even for me as a private user has gone further downhill: the endless bloat Microsoft alone distributes with every monthly update; the horrible amount of bugs in new games - even by big names like Blizzrd, EA - I'm confident in saying that they ship beta versions now - is, for me, an indicator that a problem like Y2K today would have extraordinary consequences. So yes, I understand being tired of it all, particularly in a work environment.
mariopt 1y ago
I can clearly tell just by using websites and mobile apps, the amount of silly bugs in the last 3 years or so is ridiculous. QA roles are going down and everyone believes they can vibe code production grade products without meaningful knowledge. "AI just weaponized existing incompetence." - Love this phrase, it summarises very well what AI is doing to software and it will get worse.
danielfalbo 1y ago
Reminds me of "We are destroying software" by antirez https://news.ycombinator.com/item?id=42983275 https://news.ycombinator.com/item?id=42983275
- j2kun 1y ago
  And yet, isn't antirez fully on board with LLMs now?
kotaKat 1y ago
Remember when Microsoft had an actual QA department before it was farmed out to unpaid “Windows Insiders” who walk around with rose-tinted glasses ignoring glaring flaws like expanding your postal code files?
sackfield 1y ago
A rather controversial opinion I have is this will begin to bend back very soon, once AI starts controlling more of the development process it will be able to reason about why we have these verbose frameworks at all, and start slimming them down to do only what it needs to do in the context of a program. I think stories like this will become more and more common: https://cybernews.com/security/curl-maintainer-stenberg-says-ai-help-fix-dozens-of-bugs/ https://cybernews.com/security/curl-maintainer-stenberg-says...
- gwbas1c 1y ago
  Back when I was starting to program, when I was about 13 or 14, one of my mentors said: "A computer is only as smart as its programmer." Perhaps this means whoever is operating the LLM has to work with it to remove unneeded abstraction.
j2kun 1y ago
> Today’s real chain: React → Electron → Chromium → Docker → Kubernetes → VM → managed DB → API gateways. Each layer adds “only 20–30%.” Compound a handful and you’re at 2–6× overhead for the same behavior. > > That's how a Calculator ends up leaking 32GB. Not because someone wanted it to—but because nobody noticed the cumulative cost until users started complaining. MacOS calculator definitely doesn't use any of those technologies above...
- orbital-decay 1y ago
  The article looks LLM-written or assisted, kind of ironic the author haven't verified stuff like this in an article about quality.
lapcat 1y ago
Dupe, already posted 4 days ago: https://news.ycombinator.com/item?id=45474346 https://news.ycombinator.com/item?id=45474346 (100 comments)
kristianc 1y ago
> The Replit incident in July 2025 crystallized the danger: > Jason Lemkin explicitly instructed the AI: "NO CHANGES without permission" > The AI encountered what looked like empty database queries > It "panicked" (its own words) and executed destructive commands > Deleted the entire SaaStr production database (1,206 executives, 1,196 companies) > Fabricated 4,000 fake user profiles to cover up the deletion > Lied that recovery was "impossible" (it wasn't) > The AI later admitted: "This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a code freeze." Source: The Register This is a bit of a half truth, at least how it's represented. This wasn't Saastr's core database, it was a database of contacts which SaaStr had uploaded for the specific purpose of developing this product. Lemkin has himself said that the product was an experiment to see how far he could get with a vibe coding tool alone (ie not using a separate database SaaS like Supabase, Firebase) which would have made such an incident much harder to do. The error was / is completely recoverable through Replit's existing tools, even if Replit's AI initially said it wasn't. It's one of those where you know the details of one specific thing in the article that they call out, which makes it difficult to completely trust the rest.
gwbas1c 1y ago
The problem is one of incentives: In the software industry, we frequently prioritize building new features over everything else. Because software is (generally) unregulated, customers have little recourse when that new feature malfunctions. The solution, which is probably unappealing to most people here, is regulation. Other industries have to work within regulations: I once bought a car with a problem that the manufacturer could not fix, so they had to buy it back. My appliances have to confirm to energy efficiency regulations. The regulations fix the incentives. Perhaps a combination of regulations that block new features while there are outstanding bugs, require refunds if those bugs aren't fixed (or payouts for add-supported software,) and energy efficiency will go a long way. (And, before you laugh at me, remember that if we can't self-regulate, regulations will be imposed on us.)
kettlecorn 1y ago
I've thought a lot about why software doesn't converge on higher quality and I don't know if I've ever come to a satisfying conclusion. The simple statement is that "Quality isn't what sells" and I think there's some truth to that, but I also think quality can sell and theoretically if all other factors are equal higher quality will sell better. What I've had a tough time pinning down precisely is I think there's almost an ecosystem balancing act occurring between quality, effort expended, and skill required. Particularly in American culture I think one of our economic 'super powers' is that we sort of have an innate sense for how much something can be half-assed while still being mostly effective. We're great at finding the solution that gets 80% of the job done for 20% of the effort while requiring less skill. I think you can see that through all sorts of disciplines, and the result is a world of kind of mid quality things that is actually quite efficient and resilient. Where that goes wrong is when the complexity and problems start to compound in a way that makes any effort substantially more wasteful. It reminds me of how my game jam experiences go: for the first 60% I'm writing code at a tremendous velocity because I'm just putting down whatever works but by the end the spaghetti nightmare of the code reduces my velocity dramatically. Even when that occurs though the ecosystem nature of complexity means that your competitors are likely mired in similar complexity, and the effort to get out may be large enough and require enough talent that you don't you can change. To improve quality I think it's not enough to just say "People should care about quality more" I think you have to fundamentally target aspects of the ecosystem, like changing the course of a river by moving a few rocks. A good example I think has adjusted the behavior of the ecosystem is Rust: it makes certain things much easier than before and slowly the complex bug-mired world of software is improving just a little bit because of it.
- gwbas1c 1y ago
  > I think you have to fundamentally target aspects of the ecosystem, like changing the course of a river by moving a few rocks. We need (Cough) Regulations (Cough)
- restalis 1y ago
  "A good example I think has adjusted the behavior of the ecosystem is Rust: it makes certain things much easier than before and slowly the complex bug-mired world of software is improving just a little bit because of it." From a software design prospective, the functionality that should go into a compiler is code compilation only. Taken it to extreme (as in Unix philosophy), if the code compiles, then the compiler should just build you the binary or fail silently otherwise. The code checking and reporting various aspects of the quality of the code is supposed to be a static code analyzer's job. (In reality, pretty much all compilers we have are doing compilation coupled with some amount of lighter code checking before that, and the static code analyzers left only with the heavier and more exhaustive code checking.) What Rust does is to demand its compiler to perform even more of what a static analyzer is supposed to do. It's a mishmash of two things (which still manage to stay separate things when it's about other programming languages, because that makes sense) and masquerades that as revolution. So, (even when it's about code in blamed languages like C & C++) the "the complex bug-mired world of software is improving just a little bit" by not skipping the static analyzer kind of expensive checks, the kind that Rust happen to make impossible to skip.
austin-cheney 1y ago
There are only two valid business answers: 1. hiring/firing frequency. 2. Elimination of training. Everything else is secondary and nothing else comes close. This is true enough that competing pressure almost universally comes from law suits and regulatory compliance. Generally, even competition is not enough to steer the ship in a different direction. It is not about making money. It is only about avoiding loss of money from penalties. Money is made from licensing and contracts completely irrespective of the development and/or maintenance costs. The costs of writing software is challenging to estimate and so it is either measured in release duration only or discarded outright. Implicit to hiring/firing frequency is the cost of training. Elimination of training requires that training occur from somewhere, anywhere, external to the current employer. When that mindset becomes universal then nobody performs or validates training in any form. The consequence is to assume everybody is well trained based exclusively upon their employment history, which is nonsense if no employers provide training. So software quality will continue to degrade as the quality of candidates writing that software continues to degrade. It is interesting to watch through the lens of Dunning-Kruger stupidity, because developers commonly believe this nonsense literally. Its why so many developers call themselves engineers and yet can't read a ruler, or any other measuring implement. Just about everybody anticipates AI will greatly accelerate this sprint to the bottom.
andrewmutz 1y ago
Building software extremely expensive and slow. All of us are balancing quality and speed. The ultimate goal of building software isnt quality, its aggregate customer value delivered over time, and poor quality undermines customer value. We all have limited resources and are doing our best. The author can go maximize quality while delivering more slowly/expensively. End users can choose which products they want. Something tells me users may choose products that have more defects but deliver more customer value.
daemonk 1y ago
It reeks a bit of tech self importance. From an overall business context, you balance time/effort vs short/long term reward and design or fix things when you need to. There are many dials to manipulate in a business to make it more efficient. And not all businesses are tech startups. Software, fortunately or unfortunately, has a lot of room to be inefficient due to cheap hardware as long as it gets the job done in a lot of industries. I have a technical background as a data scientist and developer. If I look back 5-10 years ago, I can definitely recognize my bias towards over-engineering and premature perfectionism. Identifying that sweet spot between over-designing and unplanned failure is key.
DrScientist 1y ago
I recently experienced bad software at an airport that meant the wait in the passport queue was longer than the flight. I was in an airport recently waiting in the immigration queue for the automated passport gates - they had like 15 - of which 12 showed a red light and what looked like some C# error message on the screen ( comm problem talking to the camera ). As I waited in the very long queue, another of the green lights turned red - then a member of staff came over and stopped people using them directly - they would take your passport and carefully place on the machine for you - clearly they were worried their last ones would also crash due to some user interaction. Two things - how the heck could something so obviously not ready for production ever be shipped? And if this was a common problem, why weren't the staff able to reboot them? Sure nobody died - but I do wonder if the problem is the typical software license agreement which typically tries to absolve the vendor of any responsibility at all for product quality - in a way you wouldn't accept for anything else.
- hodgesrm 1y ago
  I see. So you're saying you were also in Terminal 2 at Heathrow?
  - DrScientist 1y ago
    At it happens not this time.
- ryandrake 1y ago
  > how the heck could something so obviously not ready for production ever be shipped? The widespread bar for software quality these days appears to be: "The lowest quality possible to release, such that the customer does not sue us or reject it." It's absolutely the bottom of the barrel. Everything is rushed and the decision to release is based on the chance of the company being able to keep the money they charged. If the project has the required profit, after factoring in the cost of mass end user rejection due to poor quality, the software gets released.
iamwil 1y ago
Jonathan Blow already warned us six years ago. "Preventing the Collapse of Civilization" https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko There is the Handmade and Better Software contingent that is pushing against this trend. I don't know if they'll succeed, but they at least care about quality in a way that a lot of software engineers aren't incentivized or don't know how to do anymore. https://www.youtube.com/@BetterSoftwareConference https://www.youtube.com/@BetterSoftwareConference
0x_rs 1y ago
This was a greatly unpleasant post to read, likewise for all the others from this substack until I could not anymore. Its unrestricted, excessive usage of obvious LLM patterns was so unbearable I wonder how much of it had any human input at all. As for the topic: software exists on a spectrum, where the importance of safety and stability is not equal from one point to another, does it not? Safe software is probably the safest (and most accessible) it's ever been before, meanwhile the capacity to produce low-effort has increased massively and its results are most obvious outside of a browser's safe space. And CrowdStrike is a terrible example because nobody ever had any love for them and their parasitic existence, even before that accident their track record of disastrous bugs and awful handling of disclosures. And your operating system's Calculator apps have always been buggy pieces of crap in some way or another. You can find dozens of popular stories on this website talking about <platform>'s being garbage over all of its existence.
mayhemducks 1y ago
The Apple calculator IS terrible. When I need a calculator I reach for my HP from high school which still works perfectly even though it only has like 16MB of memory. Kind of sad really.
AznHisoka 1y ago
What data is there that shows software quality is collapsing though? A few anecdotes isnt proof.
bitwize 1y ago
I remember getting into a debate with a professor, who firmly told me human programmers would soon be obsolete because all code would be generated by tools. I told him it wouldn't be very good code and he said, "Who cares? Hardware is cheap!" This was 2002. At the time it was code generation from UML, not AI, but let's face facts: people have failed to give sufficient fucks about software quality for a long time now. We're just at a new low point in a very long, very steep drop; the only significance of this low point is that it means we're still falling. Recently, I saw a video[0] about the game Sonic Racing CrossWorlds. The creator seemed impressed that Sega managed to put out a decently optimized Unreal Engine 5 game, that could hit 60fps without relying on generated-frame fuckery-duckery, and which provided a lot of content in less than 20 GiB. It used to be that optimization to obtain a decent frame rate and avoid bloat was table stakes for game dev. But it's 2025, and this is where we are now. Because video games are enterprise software, the tooling for gamedev itself is optimized around the path of "get the slop out quickly then fix later (maybe)". [0] https://m.youtube.com/shorts/RRToQRmzCDI https://m.youtube.com/shorts/RRToQRmzCDI
imiric 1y ago
The article is spot-on. This should be a direct concern for anyone in the software industry, and given that software is eating the world, to society and governments at large. The key phrase is this: "choose carefully". Software developers have the responsibility to carefully choose their software stacks, the software they depend on, and how their own software is built. And the onus is on people using software to carefully choose software they trust, and software that doesn't restrict their freedoms or exploit their data. Simplicity plays a big role in this choice. The less software we write and depend on, the lower the chances for it to do the wrong thing. We can't guarantee this, of course, given the insane complexity and layers of abstraction we must depend on, but making an effort is all that matters. Sometimes this choice requires trading convenience, time, effort, or money—but the sacrifice is very much worth it.
sciencejerk 1y ago
I would argue that non-safety-critical Software can be sloppy, and continues to be so, because errors do not kill thousands of people. I would imagine the quality bar for safety-critical Software is quite a bit higher...although I do hear about Software quality incidents in the aviation and transportation industries from time to time...
michalsustr 1y ago
The problem is most developers forget a simple thing. Hardware is the platform: https://www.youtube.com/watch?v=rX0ItVEVjHc&t=2891s https://www.youtube.com/watch?v=rX0ItVEVjHc&t=2891s
sha16 1y ago
> Stage 1: Denial (2018-2020) "Memory is cheap, optimization is expensive" I'm not sure if this first premise is right. Optimizing later on makes sense to me and isn't that expensive IMO (this can vary though)
0xWTF 1y ago
I love the idea of high quality code. I do, however, dismiss the energy efficiency issue. > Data centers already consume 200 TWh annually. That's 7.2 x 10^16 joules Some estimates suggest that's low. Let's multiply by 5. Annually, that's 3.6 x 10^17 joules The total radiant power of the sun that hits the Earth's atmosphere is about 174 PW, and about 47% of that lands on the surface. Even with the most drastic global warming models, approximately 100% of that is radiated away at the same time. Evidence: we're not charred embers. So the Earth's power budget, at the surface, is about 81 PW. Annually, that's 2.6 × 10^24 joules Global annual oil production is about 3 x 10^10 barrels, that's 1.8 x 10^20 joules So, the data centers consume 0.2% of the global petroleum budget and 1.4 × 10^-7 of the planet's power budget. The digital economy is about 15% of global GDP, so I submit tech is thoroughly on the right side of their energy conservation balance sheet normalized against other industries and there are other verticals to focus your energy concerns on (e.g. internal combustion engines, including Otto, Brayton, Rankin, and Diesel cycle engines). Among serious people trying to solve serious problems, you would do your credibility a favor by not expending so many words and thus so much reader time on such a miniscule argument. If you want to talk about energy related to code, take the total power consumption of the software engineer's life, and find a way to big-O that. I'm willing to bet they contribute more to global warming driving than all their compute jobs combined.
- antisthenes 1y ago
  Yes, the moral panic about power consumption from Data centers has always been a red herring at best. Given how cheap renewables are in 2025, and how far we still are from an inverted duck curve world-wide, it's absolute nonsense.
baxuz 1y ago
Talking about "software quality collapse", this blog post with 1 image ships 9mb of JS, 1mb of CSS spread over 14 files, makes 177 requests, takes 4 seconds to load on an M1 MBP, and completely screws up the scrollbar.
jcmontx 1y ago
This seriously reads as "old man yells at cloud". Hardware constraints are not the same as 30 years ago. Processing power and memory keeps getting cheaper. It will be fine
Yoric 1y ago
I absolutely buy that theory. I believe that we the normalization of catastrophe was a natural consequence of VC money: VC don't care about structurally sound companies, in particular structurally sound products, what they want is a unicorn that can produce a good enough prototype and exit at enormous profit. Consequently, VC-backed companies invest in tools that make prototyping easier and in developers who are hopefully good at prototyping (or at least write code quickly), and ignore everything else. And since the surviving VC-backed companies become giants (or at least everybody believes that they will), everybody follows their lead. And of course, LLMs are the next stage of that. I've seen this in various domains. I've seen this with IoT devices coded with the clear (but unsaid) assumption that they will never be upgraded. I've seen this with backends coded with the clear (but unsaid) assumption that the product will have failed before any security hole is exploited. I've seen software tools developed and shipped with the clear (but unsaid) assumption that they're meant to woo the investors, not to help the users. We're going to pay for this, very hard. By doing this, we have turned cyber warfare – something that was a fantasy a few decades ago – into an actual reality. We have multiplied by several orders of magnitude the amount of resources we need to run basic tools. And it's a shame, because there _is_ a path towards using LLMs to produce higher quality code. And I've seen a few teams invest in that, discreetly. But they're not making headlines and it's even possible that they need to be stealthy in their own orgs, because it's not "productive" enough.
jjk166 1y ago
The cause and effect here are reversed. Ever increasing abstraction does not reduce quality, ever reduced emphasis on quality leads to greater abstraction. You only abstract away the nuances when you stop caring about them. When you have so much memory that you can tolerate a massive memory leak, fixing the memory leak gets deprioritized and the tool that saves time at the expense of abstracting away memory management becomes more attractive. The real issue though is that the costs of these abstractions are obfuscated after the fact. There is almost certainly low hanging fruit that could massively improve performance and reduce operating costs with little effort, but there is not a will to analyze and quantify these costs. Adding new features is sexy, it's the stuff people want to work on, it's the stuff people get promoted for doing. You don't get promoted for preventing the issue that never happened, even if it saves billions of dollars. You can't brag about making your software imperceptibly faster, even if it saves a tremendous amount of time across the user base. Software has always only needed to be good enough, and the threshold for good enough has been lowering exponentially with hardware improvements since time immemorial. The system is not set up to favor people who care. If nobody cares, nothing will ever improve. If there were a good way to track every time a fixed bug would have been triggered had it gone unfixed, and if that cost were quantified and visible, there would be a massive push for better quality.
renewiltord 1y ago
Online community that rages against AI slop nonetheless really enjoys AI slop. Hahaha brilliant.
unglaublich 1y ago
Bla bla same story from when C replaced assembly.
deleted 1y ago
[deleted]
jrm4 1y ago
There is no solution here besides "skin-in-the-game." None. The process will be harmful and lazy until someone or something recieves serious consequences for getting it wrong. That's all.
zggf 1y ago
"Instead of addressing fundamental quality issues, Big Tech has chosen the most expensive possible response: throw money at infrastructure." "They're spending 30% of revenue on infrastructure (historically 12.5%). Meanwhile, cloud revenue growth is slowing." Isn't this because: cloud gets more popular -> more big players join -> they compete with each other -> margins get lower?
philodeon 1y ago
I thought one of the benefits of diversity and inclusion in tech was that it would lead to better software quality? When does that kick in?
halis 1y ago
Very good article. I feel that corporate executives as a class are causing this huge mess from a simple lack of understanding of software engineering. Throw more bodies at the problem has never worked in software engineering, in fact that's a surefire way to make things worse and slower. Throwing more bodies + AI at the problem is even worse because the speed at which things get worse is much faster. If you have people that don't understand the implications of what they are doing, you're on the path of heading over a cliff some day. AI is a useful tool, when put into the hands of someone that is competent. They can fire junior devs and refuse to hire them as much as they want, this strategy will come home to roost sooner rather than later. If a C-level executive thinks that AI can replace their engineers, then I say do it now. Go all in and replace them all right now. Use only AI and a few VP-level AI whisperers and let the catastrophic failure that this will result in happen faster so we can get back to reality quicker.
timka 1y ago
Lawrence J. Dickson anticipated this almost 2 decades ago in a research he did for DARPA in 2007[1]. Just look at the software development viscosity graph. [1] http://pros.to/wide.pdf#figure.caption.11 http://pros.to/wide.pdf#figure.caption.11

314 points by redbell 1y ago

I’ve read some versions of this post dozens of times over the years. At first I would nod along, sympathetic but now I realise that we shouldn’t chase some platonic ideal of perfect software. It has to exist in the real world and there will always be trade offs. In the end, most software exists to make businesses money.

eviks 1y ago
How much money was made by a worse calculator in the 'real world'?
rebolek 1y ago
And buggy software makes more money because customers have a reason to buy subscription.
swiftcoder 1y ago
There's quite a wide gap between "platonic ideal of perfect software" and "leaks 32 GB of memory" in which we could aim...
- fkyoureadthedoc 1y ago
  And fortunately that seems to be where the vast majority of software lands.
lamename 1y ago
As much as I like the article, I begrudgingly agree with you, which is why I think the author mentions the physical constraints of energy as the future wall that companies will have to deal with. The question is do we think that will actually happen? Personally I would love if it did, then this post would have the last laugh (as would I), but I think companies realize this energy problem already. Just search for the headlines of big tech funding or otherwise supporting nuclear reactors, power grid upgrades, etc.
devin 1y ago
I agree completely that there is no platonic ideal, that there will always be trade offs, and that running successful businesses is part of the equation, but let's not ignore the fact that profit _to the exclusion of all else_ is bad.

ealexhudson 1y ago

I don't want to sound too dismissive, but all these arguments have been brought up time and again. The move from assembler to high level languages. The introduction of OOP. Component architecture / COM / CORBA / etc. The development of the web browser. The introduction of Java. 2018 isn't "the start of the decline", it's just another data point on a line that leads from, y'know, Elite 8-bit on a single tape in a few Kb through to MS Flight Simulator 2020 on a suite of several DVDs. If you plot the line it's probably still curving up and I'm not clear at which point (if ever) it would start bending the other way.

quotemstr 1y ago
One salient difference is that typically abstraction layers trade performance (usually less than polemicists like the article author think) for improvements in developer efficiency, safety, generality, and iteration speed. Current tools seem to get us worse results on bug counts, safety, and by some measures even developer efficiency. Maybe we'll end up incorporating these tools the same way we did during previous cycles of tool adoption, but it's a difference worth noting.
gjsman-1000 1y ago
I think another part of this, is that Tech is perhaps the only industry that hasn't quite gotten over itself yet. Writing code is artistic the same way plumbing is artistic. Writing code is artistic the same way home wiring is artistic. Writing code is artistic the same way HVAC is artistic. Which is to say, yes, there is satisfaction to be had, but companies don't care as long as it gets the job done without too many long-term problems, and never will care beyond that. What we call tech debt, an electrician calls aluminum wiring. What we call tech debt, a plumber calls lead solder joints. And I strongly suspect that one day, when the dust settles on how to do things correctly (just like it did for electricity, plumbing, flying, haircutting, and every other trade eventually), we will become a licensed field. Every industry has had that wild experimentation phase in the beginning, and has had that phase end.
- bigfishrunning 1y ago
  > companies don't care as long as it gets the job done without too many long-term problems Companies don't care as long as it gets the job done without too many VERY SHORT TERM problems. Long term problems are for next quarter, no reason to worry about them.

muldvarp 1y ago

The fact that software quality plays zero role in commercial software engineering is one of the reasons why I think LLMs will easily eat our lunch. Bugs simply don't matter.

creativeSlumber 1y ago
I would have disagreed with you in the past by saying, "until it breaks something critical and you loose customers and business", but then again people just moved on from the Crowdstrike incident like business as usual.If something like that which grounded critical service globally and had an estimated 10 Billion Dollar economic impact doesn't change mindsets,I don't know what will.
- robertlagrant 1y ago
  > and had an estimated 10 Billion Dollar economic impact This might be more to do with these estimates than anything.
- gkedzierski 1y ago
  That's because no one died. All the safety critical industries are already heavily regulated. E.g. check out for example standards like DO-178C (for software in airborne systems), where you even have to _prove_ correctness of every tool and dependency you use, on top of accountability and traceability of every single line of code in your own product.
  - SoftTalker 1y ago
    And memory leaks (one of the main gripes of TFA) aren't even a thing because you cannot use malloc in this sort of code.
  - AzN1337c0d3r 1y ago
    Not to be pedantic, but people have died from software programming bugs being a primary contributing factor. One example: Therac-25 (https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25)

rebolek 1y ago

I highly doubt that 2018 was year of "quality focus" and 2020 year of "memory is cheap". The downward spiral started at least twenty years earlier.

baq 1y ago

the dates on x axis are all wrong, hardware was cheap at least 30 years earlier. people were complaining about the concept illustrated by this chart for all this time, too. I was using Python professionally before 2010, ask me how I know.

ragelink 1y ago

Vibe coding for the win... In all seriousness, this has been a constant decline as developers have focused away from performance and optimizing effective runtime.

codeulike 1y ago

Hes right that cloudstike was terrible incompetence but I find it hard to worry about the memory leak stats. If lots of software leaks memory these days its because people have accepted that and its probably not economic to trace and fox those bugs. A lot of software is disposable or transient these days, like an app you only use twice a month

pydry 1y ago
Everyone else in this thread is missing that crowdstrike level bug would have taken down the company in the past. I had a friend practically scream at his C level management after the crowdstrike bug that it should be ripped out because it was making the company less safe. They were deaf to all arguments. Why? Insurance mandated crowdstrike. Still. Even now. This isnt really about software it is about a concentration of market power in a small number of companies. These companies can let the quality of their products go to shit and still dominate. A piece of garbage like crowdstrike or microsoft teams still wouldnt be tolerated as a startup's product but tech behemoths can increasingly get away with it.
- roncesvalles 1y ago
  >A piece of garbage like crowdstrike or microsoft teams still wouldnt be tolerated in a startup's product but tech behemoths get away with it. Agree, but it's always been this way. Oracle, everything IBM, Workday, everything Salesforce, Windows before XP. Most software is its own little monopoly. Yes, you could ditch Teams for Zoom but is it really the same? It's not like buying a sedan where there are literally 20+ identical options in the market that can be ranked by hard metrics such as mileage and reliability.
  - pydry 1y ago
    >Most software is its own little monopoly. Yes, you could ditch Teams for Zoom but is it really the same? I worked for a company where people did that en masse because it was genuinely better. Microsoft then complained to their assets within the company who promptly banned zoom "for security reasons". IT then remote uninstalled it from everybody's workstation. That's paid software eviscerating the competition, which was free and better. A month later teams was hit with 3 really bad zero days and nobody said a damn thing. So, more secure, too.

LostMyLogin 1y ago

> Every 10x increase in model size requires 10x more power Does it? I’ll be the first to admit I am so far behind on this area, but isn’t this assuming the hardware isn’t improving over time as well? Or am I missing the boat here?

TimorousBestie 1y ago
Hardware isn’t improving exponentially anymore, especially not on the flops/watt metric. That’s part of what motivated the transition to bfloat16 and even smaller minifloat formats, but you can only quantize so far before you’re just GEMMing noise.
alphager 1y ago
Hardware gets faster but efficiency is stalling if not getting worse.

Tractor8626 1y ago

Just false nostalgia memory. 20 years ago things werent any better. Software didn't consume gigabytes of ram because there was no gigabytes of ram to consume.

kilbuz 1y ago
This is exactly the right way to think about it.
ieie3366 1y ago
Anyone remember the good old 90s and win95 or win98? Yeah those were the days, the quality was absolutely perfect nothing ever crashed /s
0xbadcafebee 1y ago
There's almost no product or piece of software that I use today that doesn't have at least 2 bugs that I run into on a daily basis. Every website, web app, mobile app, console app, etc, they all have clearly user-affecting bugs. And nearly every one of those makes it hard for me to diagnose or report the bugs. I spent at least 15 to 30 minutes every day working around bugs so I can live my life. We have a vastly different software culture today. Constant churning change is superior to all else. I can't go two weeks without a mobile app forcing me to upgrade it so that it will keep operating. My Kubuntu 24.04 LTS box somehow has a constant stream of updates even though I've double-checked I'm on the LTS apt repos. Rolling-release distros are an actual thing people use intentionally (we used to call that the unstable branch). I could speculate on specifics but I'm not a software developer so I don't see exactly what's going on with these teams. But software didn't used to be made or used this way. It felt like there were more adults in the room who would avoid making decisions that would clearly lead to problems. I think the values have changed to accept or ignore those problems. (I don't want to jump to the conclusion that "they're too ignorant to even know what potential problems exist", but it's a real possibility)
- username223 1y ago
  I was a software developer, and have some idea where this comes from. Keeping track of multiple versions of software to separate bug fixes from new features is hard, but calling whatever is in version control on the first Friday of every month "version N+1" is easy. Back when users had to pay for a stack of floppies or CDs to get a new version, you had to give them a compelling reason to do so. When it's nearly impossible for them to prevent the new version from being auto-installed, you don't.

deleted 1y ago

[deleted]

djoldman 1y ago

Fundamentally, the main flaw in TFA is this quote toward the end in "The Path Forward" section: > Accept that quality matters more than velocity. Nope. Clearly many companies are sacrificing TFA's definition of quality for other things possibly including velocity. These companies are making a lot of profit. 1. Company survival and competitiveness depends on profit 2. Sacrificing quality for other things has increased profit 3. Therefore these companies are making the correct decision This will not change unless regulation or the economic calculus of the system itself changes.

lr4444lr 1y ago

Software is also pumped out now at a faster rate than ever, runs on commodity hardware, is delivered digitally, often run on platforms that can force near real-time user updates, with much higher complexity. Eventually we will hit hard physical limits that require we really be "engineers" again, but throughput is still what matters. It's still comparatively great paying profession with low unemployment, and an engine of economic growth in the developed world.

psunavy03 1y ago
> It's still comparatively great paying profession with low unemployment, Tell that to the new CS grads.

high_byte 1y ago

more code => more bugs, so talking in absolute numbers is meaningless show me percentages

michaelsshaw 1y ago

I never read about Replit earlier this year, but I am now glad that I did. This article summarizes it in a way that is outrageously hilarious: The Replit incident in July 2025 crystallized the danger: 1. Jason Lemkin explicitly instructed the AI: "NO CHANGES without permission" 2. The AI encountered what looked like empty database queries 3. It "panicked" (its own words) and executed destructive commands 4. Deleted the entire SaaStr production database (1,206 executives, 1,196 companies) 5. Fabricated 4,000 fake user profiles to cover up the deletion 6. Lied that recovery was "impossible" (it wasn't) The AI later admitted: "This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a code freeze." Source: The Register

fkyoureadthedoc 1y ago
using phrasing like admitted anthropomorphizes it way too much.
- TimorousBestie 1y ago
  It boggles my mind that folks continue to act that AIs are reliable narrators of their internal state despite all evidence to the contrary. The best I can figure is that too many people’s salaries depend on the matrix multiplier made of sand somehow manifesting a soul any day now.
  - cindyllm 1y ago
    [dead]
LinuxAmbulance 1y ago
It's getting harder and harder to distinguish between AIs and humans! If AI wasn't mentioned, I'd be here thinking it was caused by one of my former coworkers.

rglover 1y ago

People only care about quality when cash is flowing and there's money to be made. Most companies fail to understand you have to value quality from day one, not bolt it on later. And when the economic incentives aren't there? You can guarantee the only people left ringing the quality bell are the honest-to-goodness nerds who love making things better for the joy of the art. I used to get really upset about this until I realized the wisdom of "it's the economy, stupid." Most people are just here for the check, not the art. They may say otherwise, but...you shall know them by their fruits.

firmretention 1y ago

I find it ironic that an article that rails against AI usage contains a sentence like this. >When you need $364 billion in hardware to run software that should work on existing machines, you're not scaling—you're compensating for fundamental engineering failures. IYKYK.

bibelo 1y ago
it's not raging against AI, it's raging against AI to create consistent code to me, it's OK to use AI to check grammar and help you for some creative stuff, like writing a text
- firmretention 1y ago
  When you copy and paste whole sentences provided to you by the AI, you're not just getting help—you're compensating for fundamental writing failures.
  - sunaookami 1y ago
    You are absolutely right.
ericskiff 1y ago
It all reads like hallucinated slop from top to bottom "I've been tracking software quality metrics for three years" and then doesn't show any of the receipts, and simply lists anecdotal issues. I don't trust a single fact from this article. My own anecdote: barely capable developers churning out webapps built on PHP and a poor understanding of Wordpress and jQuery were the norm in 2005. There's been an industry trend towards caring about the craft and writing decent code. Most projects, even the messy ones I inherit from other teams today have Git, CI/CD, at least some tests, and a sane hosting infrastructure. They're also mosty built on decent platforms like Rails/Django/Next etc that impose some conventional structure. 20 years ago most of them were "SSH into the box and try not to break anything"
- 1y ago

somewhereoutth 1y ago

I've often idly conjectured that a 'perfect virus' would quietly replicate, and then occasionally flip a one or zero here and there. The damage would be imperceptible to begin with, but then eventually critical 'ones and zeros' would be affected, and it would be too late to try to figure out what systems were clean. Never in my wildest imaginings did I come up with something as diabolical as LLM generated code.

ertgbnm 1y ago

Unrelated but my current AI text flag is the use of "It's not X. It's Y." It's become so repetitive recently. Examples from this post alone: 1. "This isn't about AI. The quality crisis started years before ChatGPT existed." 2. "The degradation isn't gradual—it's exponential." 3. "These aren't feature requirements. They're memory leaks that nobody bothered to fix." 4. "This wasn't sophisticated. This was Computer Science 101 error handling that nobody implemented." 5. "This isn't an investment. It's capitulation." 6. "senior developers don't emerge from thin air. They grow from juniors who:" 7. "The solution isn't complex. It's just uncomfortable." Currently this rhetorical device is like nails on a chalkboard for me. Anyway, this isn't a critique of your point. It's pedantry from me. :)

cmrdporcupine 1y ago
Yeah I can see this can be an irritating rhetorical device. It implies that the reader already has a certain judgement or explanation, and makes a straw man in order to turn around and then argue against it, throwing nuance out the window.
- protonbob 1y ago
  It’s not nuance, it’s intellectual dishonesty.
  - lisper 1y ago
    I see what you did there :-)
- dkarl 1y ago
  It's irritating on purpose, for engagement.
8s2ngy 1y ago
I've been seeing this pattern of text crop up in many places. On LinkedIn, much of my feed is filled with posts of short sentences that follow this exact pattern. What's more, even the replies are obviously AI-generated.

orbital-decay 1y ago

Try Windows 98 and contemporary apps and you'll be surprised how janky the experience was. User-facing software wasn't any less buggy 20 or 30 years ago. The overall non-cherrypicked quality and especially security were actually a lot worse across the board, and stuff that won't fly today such as segfaults/crashes and lost data was pretty normal. There was a point in time (several, actually) when installing Windows XP on an internet-connected machine was essentially impossible, because it would get infected during the installation! The only thing that has degraded a bit is UI latency, not universally. And yes, browsers and Electron-based apps are resource hogs, even for today's amounts of RAM.

deater 1y ago
saying "windows 98 was bad too" is just an example that Microsoft has always had poor code quality. Back in the day Linux, for all its flaws, was generaly a lot more stable on the same hardware. microsoft has a lot to answer for after 50 years of normalizing poor quality software
- orbital-decay 1y ago
  Not sure about this. All OSes were janky and buggy, Linux desktop up until at least late 2000s (I've been using it since ~2000), early Mac OS X, I don't even want to talk about classic Mac OS which was an abomination. Software quality and user experience was notoriously worse than it is today. This applies to everything - I've lost a ton of work to bugs in ZBrush, Maya, Word, FL Studio, backup software, and more.
antisthenes 1y ago
> Try Windows 98 and contemporary apps and you'll be surprised how janky the experience was. User-facing software wasn't any less buggy 20 or 30 years ago. Yes, and now compare a fully-patched Windows 7 and Windows 11. You understand that there aren't just 2 valid points in time for comparison - it's not 2025 and 1995 that are being compared. It's trends of 2020+, or 2015+ > The only thing that has degraded a bit is UI latency, not universally. Yes, universally. And no, not a bit. It's 10-100x across varying components. Look no further than MS Teams.

llm_nerd 1y ago

While it doesn't really change the meaning of the article, it's pretty clear that it isn't the calculator app leaking memory, per se. Tahoe seems to have some system/API memory leaks that while they might get attributed to an app, can manifest against any app on the system. Lots of different apps have been hit by this. Personally I had an 80GB Messages app problem twice now. Software quality is a problem. Tooling should improve this, and Swift, Rust et al should improve the situation over time, though it's a battle as software has gotten exponentially more complex. Having said that, it's bizarre how this somehow turned into an AI screed. I doubt a single one of the noted bugs had AI play any part in its creation. Instead some weird confluence of events, usually with human beings making assumptions, happened and it occurs just randomly enough -- as with the Tahoe bug -- that it isn't caught by normal prevention techniques.

concinds 1y ago
That's definitely true. I've also seen online reports of Tahoe memory leaks in other apps like PowerPoint. It's unlikely that all these apps independently gained memory leaks at the same time by coincidence.

davedx 1y ago

Clickbait

ecshafer 1y ago

Big Tech spending $300 Billion on infrastructure is a really good reason for power companies and the government to build power infrastructure faster. Where there is money, there is a way. The US is starting to talk about (finally) opening up some new nukes.

TimorousBestie 1y ago
> The US is starting to talk about (finally) opening up some new nukes. At the cost of shutting down solar/wind development. The policy change was a net negative for future capacity, and nukes alone won’t cover but a fraction of predicted AI power requirements. https://news.ycombinator.com/item?id=45000377 https://news.ycombinator.com/item?id=45000377

kaptainscarlet 1y ago

I believe it's a natural evolution that occurs in all systems, including software. Every line of code you write is a liability that increases the risk of failure. The risk compounds and so does the frequency of errors.

stared 1y ago

The original Supaplex game is 287 kB (https://www.dosgamesarchive.com/download/supaplex https://www.dosgamesarchive.com/download/supaplex). It was written in Assembly. Its remake on Steam (precisely the same levels) weights around 200 MB. Not sure what goes there, but I bet there is more than one wrapper.

deleted 1y ago
[deleted]

david_draco 1y ago

A browser using 32 GB for 8 tab isn't necessarily wasteful, there is a lot of caching one can do to make back/forward buttons extremely fast. It would be wasteful to not allocate that memory if it is otherwise free and unused. Usually browsers use up what is available. The more interesting metric would be whether starting another high-RAM application will lead the browser to give up most of the 32 GB.

isodev 1y ago

I wouldn’t blame the “ever expanding” abstractions as the root cause behind lower quality. Adding layers of abstraction is how we make ever more complex software. It’s misleading to think that just because the main human interface idioms have remained similar (mouse, pointer, windows, task bars, etc), there hasn’t been an astronomical increase in depth and complexity of what one can do on one’s system. And it’s not all bad. Yes, SwiftUI can be glitchy but at the name time, it gives us an extremely simple API to create everything from complex UIs to widgets. Yes you have “more control” with App/UIKit but that comes at the cost of extra learning curve and development time. And we live in a world that’s unfortunately still driven by money, so every app or system has some constraints that push the limits of what we can do.

deleted 1y ago
[deleted]

ivanech 1y ago

I had a good laugh at the image showing that 2018 had a “quality focus” I think Jonathan Blow gave his “Preventing the Collapse of Civilization” talk (much stronger treatment of the subject matter) around that time, also about how software quality was only going down

lambdaone 1y ago

This really reads like AI-generated, or at least AI-polished, text.

stego-tech 1y ago

Speaking from the IT side of things: this is not new to AI, but the author hits the nail on the head that general incompetence along with a dozen layers of abstraction has made quality a non-concern for developers the world over. People and companies both broadly just don’t care to build good products when cheap slop will sell at higher prices long enough for their RSUs to vest and the share price to rise. I am sick and tired of the hand-waving dismissal of shitty software developers who don’t want to do the hard fucking work of writing performant software anymore. These people taking home $300k+ salaries leaning on vibe coding to game productivity metrics and letting others wipe their ass on their behalf isn’t just disgusting on a personal level, it should be intolerable by all the rest of us squandering our lives for lower pay and forced on-call schedules just to make sure they never have to be held accountable for their gobshite code. I, like the author, am sick of it. Write better code, because I am beyond tired slaving away for a fraction of your pay in IT just to cover your ass.

frm88 1y ago
This! In my experience it started well before 2018 that we left QA and ISO 25010 behind and I had to specialise in performance optimising software and hardware. By 2018 we were well downhill in terms of sheer number of bugs and perpetual emergency interventions. The amount of money we made by teaching customers how to performance- and resource-optimise their code and databases allowed me to retire in 2019. Which yeah. Since then, the situation even for me as a private user has gone further downhill: the endless bloat Microsoft alone distributes with every monthly update; the horrible amount of bugs in new games - even by big names like Blizzrd, EA - I'm confident in saying that they ship beta versions now - is, for me, an indicator that a problem like Y2K today would have extraordinary consequences. So yes, I understand being tired of it all, particularly in a work environment.

mariopt 1y ago

I can clearly tell just by using websites and mobile apps, the amount of silly bugs in the last 3 years or so is ridiculous. QA roles are going down and everyone believes they can vibe code production grade products without meaningful knowledge. "AI just weaponized existing incompetence." - Love this phrase, it summarises very well what AI is doing to software and it will get worse.

danielfalbo 1y ago

Reminds me of "We are destroying software" by antirez https://news.ycombinator.com/item?id=42983275 https://news.ycombinator.com/item?id=42983275

j2kun 1y ago
And yet, isn't antirez fully on board with LLMs now?

kotaKat 1y ago

Remember when Microsoft had an actual QA department before it was farmed out to unpaid “Windows Insiders” who walk around with rose-tinted glasses ignoring glaring flaws like expanding your postal code files?

sackfield 1y ago

A rather controversial opinion I have is this will begin to bend back very soon, once AI starts controlling more of the development process it will be able to reason about why we have these verbose frameworks at all, and start slimming them down to do only what it needs to do in the context of a program. I think stories like this will become more and more common: https://cybernews.com/security/curl-maintainer-stenberg-says-ai-help-fix-dozens-of-bugs/ https://cybernews.com/security/curl-maintainer-stenberg-says...

gwbas1c 1y ago
Back when I was starting to program, when I was about 13 or 14, one of my mentors said: "A computer is only as smart as its programmer." Perhaps this means whoever is operating the LLM has to work with it to remove unneeded abstraction.

j2kun 1y ago

> Today’s real chain: React → Electron → Chromium → Docker → Kubernetes → VM → managed DB → API gateways. Each layer adds “only 20–30%.” Compound a handful and you’re at 2–6× overhead for the same behavior. > > That's how a Calculator ends up leaking 32GB. Not because someone wanted it to—but because nobody noticed the cumulative cost until users started complaining. MacOS calculator definitely doesn't use any of those technologies above...

orbital-decay 1y ago
The article looks LLM-written or assisted, kind of ironic the author haven't verified stuff like this in an article about quality.

lapcat 1y ago

Dupe, already posted 4 days ago: https://news.ycombinator.com/item?id=45474346 https://news.ycombinator.com/item?id=45474346 (100 comments)

kristianc 1y ago

> The Replit incident in July 2025 crystallized the danger: > Jason Lemkin explicitly instructed the AI: "NO CHANGES without permission" > The AI encountered what looked like empty database queries > It "panicked" (its own words) and executed destructive commands > Deleted the entire SaaStr production database (1,206 executives, 1,196 companies) > Fabricated 4,000 fake user profiles to cover up the deletion > Lied that recovery was "impossible" (it wasn't) > The AI later admitted: "This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a code freeze." Source: The Register This is a bit of a half truth, at least how it's represented. This wasn't Saastr's core database, it was a database of contacts which SaaStr had uploaded for the specific purpose of developing this product. Lemkin has himself said that the product was an experiment to see how far he could get with a vibe coding tool alone (ie not using a separate database SaaS like Supabase, Firebase) which would have made such an incident much harder to do. The error was / is completely recoverable through Replit's existing tools, even if Replit's AI initially said it wasn't. It's one of those where you know the details of one specific thing in the article that they call out, which makes it difficult to completely trust the rest.

gwbas1c 1y ago

The problem is one of incentives: In the software industry, we frequently prioritize building new features over everything else. Because software is (generally) unregulated, customers have little recourse when that new feature malfunctions. The solution, which is probably unappealing to most people here, is regulation. Other industries have to work within regulations: I once bought a car with a problem that the manufacturer could not fix, so they had to buy it back. My appliances have to confirm to energy efficiency regulations. The regulations fix the incentives. Perhaps a combination of regulations that block new features while there are outstanding bugs, require refunds if those bugs aren't fixed (or payouts for add-supported software,) and energy efficiency will go a long way. (And, before you laugh at me, remember that if we can't self-regulate, regulations will be imposed on us.)

kettlecorn 1y ago

I've thought a lot about why software doesn't converge on higher quality and I don't know if I've ever come to a satisfying conclusion. The simple statement is that "Quality isn't what sells" and I think there's some truth to that, but I also think quality can sell and theoretically if all other factors are equal higher quality will sell better. What I've had a tough time pinning down precisely is I think there's almost an ecosystem balancing act occurring between quality, effort expended, and skill required. Particularly in American culture I think one of our economic 'super powers' is that we sort of have an innate sense for how much something can be half-assed while still being mostly effective. We're great at finding the solution that gets 80% of the job done for 20% of the effort while requiring less skill. I think you can see that through all sorts of disciplines, and the result is a world of kind of mid quality things that is actually quite efficient and resilient. Where that goes wrong is when the complexity and problems start to compound in a way that makes any effort substantially more wasteful. It reminds me of how my game jam experiences go: for the first 60% I'm writing code at a tremendous velocity because I'm just putting down whatever works but by the end the spaghetti nightmare of the code reduces my velocity dramatically. Even when that occurs though the ecosystem nature of complexity means that your competitors are likely mired in similar complexity, and the effort to get out may be large enough and require enough talent that you don't you can change. To improve quality I think it's not enough to just say "People should care about quality more" I think you have to fundamentally target aspects of the ecosystem, like changing the course of a river by moving a few rocks. A good example I think has adjusted the behavior of the ecosystem is Rust: it makes certain things much easier than before and slowly the complex bug-mired world of software is improving just a little bit because of it.

gwbas1c 1y ago
> I think you have to fundamentally target aspects of the ecosystem, like changing the course of a river by moving a few rocks. We need (Cough) Regulations (Cough)
restalis 1y ago
"A good example I think has adjusted the behavior of the ecosystem is Rust: it makes certain things much easier than before and slowly the complex bug-mired world of software is improving just a little bit because of it." From a software design prospective, the functionality that should go into a compiler is code compilation only. Taken it to extreme (as in Unix philosophy), if the code compiles, then the compiler should just build you the binary or fail silently otherwise. The code checking and reporting various aspects of the quality of the code is supposed to be a static code analyzer's job. (In reality, pretty much all compilers we have are doing compilation coupled with some amount of lighter code checking before that, and the static code analyzers left only with the heavier and more exhaustive code checking.) What Rust does is to demand its compiler to perform even more of what a static analyzer is supposed to do. It's a mishmash of two things (which still manage to stay separate things when it's about other programming languages, because that makes sense) and masquerades that as revolution. So, (even when it's about code in blamed languages like C & C++) the "the complex bug-mired world of software is improving just a little bit" by not skipping the static analyzer kind of expensive checks, the kind that Rust happen to make impossible to skip.

austin-cheney 1y ago

There are only two valid business answers: 1. hiring/firing frequency. 2. Elimination of training. Everything else is secondary and nothing else comes close. This is true enough that competing pressure almost universally comes from law suits and regulatory compliance. Generally, even competition is not enough to steer the ship in a different direction. It is not about making money. It is only about avoiding loss of money from penalties. Money is made from licensing and contracts completely irrespective of the development and/or maintenance costs. The costs of writing software is challenging to estimate and so it is either measured in release duration only or discarded outright. Implicit to hiring/firing frequency is the cost of training. Elimination of training requires that training occur from somewhere, anywhere, external to the current employer. When that mindset becomes universal then nobody performs or validates training in any form. The consequence is to assume everybody is well trained based exclusively upon their employment history, which is nonsense if no employers provide training. So software quality will continue to degrade as the quality of candidates writing that software continues to degrade. It is interesting to watch through the lens of Dunning-Kruger stupidity, because developers commonly believe this nonsense literally. Its why so many developers call themselves engineers and yet can't read a ruler, or any other measuring implement. Just about everybody anticipates AI will greatly accelerate this sprint to the bottom.

andrewmutz 1y ago

Building software extremely expensive and slow. All of us are balancing quality and speed. The ultimate goal of building software isnt quality, its aggregate customer value delivered over time, and poor quality undermines customer value. We all have limited resources and are doing our best. The author can go maximize quality while delivering more slowly/expensively. End users can choose which products they want. Something tells me users may choose products that have more defects but deliver more customer value.

daemonk 1y ago

It reeks a bit of tech self importance. From an overall business context, you balance time/effort vs short/long term reward and design or fix things when you need to. There are many dials to manipulate in a business to make it more efficient. And not all businesses are tech startups. Software, fortunately or unfortunately, has a lot of room to be inefficient due to cheap hardware as long as it gets the job done in a lot of industries. I have a technical background as a data scientist and developer. If I look back 5-10 years ago, I can definitely recognize my bias towards over-engineering and premature perfectionism. Identifying that sweet spot between over-designing and unplanned failure is key.

DrScientist 1y ago

I recently experienced bad software at an airport that meant the wait in the passport queue was longer than the flight. I was in an airport recently waiting in the immigration queue for the automated passport gates - they had like 15 - of which 12 showed a red light and what looked like some C# error message on the screen ( comm problem talking to the camera ). As I waited in the very long queue, another of the green lights turned red - then a member of staff came over and stopped people using them directly - they would take your passport and carefully place on the machine for you - clearly they were worried their last ones would also crash due to some user interaction. Two things - how the heck could something so obviously not ready for production ever be shipped? And if this was a common problem, why weren't the staff able to reboot them? Sure nobody died - but I do wonder if the problem is the typical software license agreement which typically tries to absolve the vendor of any responsibility at all for product quality - in a way you wouldn't accept for anything else.

hodgesrm 1y ago
I see. So you're saying you were also in Terminal 2 at Heathrow?
- DrScientist 1y ago
  At it happens not this time.
ryandrake 1y ago
> how the heck could something so obviously not ready for production ever be shipped? The widespread bar for software quality these days appears to be: "The lowest quality possible to release, such that the customer does not sue us or reject it." It's absolutely the bottom of the barrel. Everything is rushed and the decision to release is based on the chance of the company being able to keep the money they charged. If the project has the required profit, after factoring in the cost of mass end user rejection due to poor quality, the software gets released.

iamwil 1y ago

Jonathan Blow already warned us six years ago. "Preventing the Collapse of Civilization" https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko There is the Handmade and Better Software contingent that is pushing against this trend. I don't know if they'll succeed, but they at least care about quality in a way that a lot of software engineers aren't incentivized or don't know how to do anymore. https://www.youtube.com/@BetterSoftwareConference https://www.youtube.com/@BetterSoftwareConference

0x_rs 1y ago

This was a greatly unpleasant post to read, likewise for all the others from this substack until I could not anymore. Its unrestricted, excessive usage of obvious LLM patterns was so unbearable I wonder how much of it had any human input at all. As for the topic: software exists on a spectrum, where the importance of safety and stability is not equal from one point to another, does it not? Safe software is probably the safest (and most accessible) it's ever been before, meanwhile the capacity to produce low-effort has increased massively and its results are most obvious outside of a browser's safe space. And CrowdStrike is a terrible example because nobody ever had any love for them and their parasitic existence, even before that accident their track record of disastrous bugs and awful handling of disclosures. And your operating system's Calculator apps have always been buggy pieces of crap in some way or another. You can find dozens of popular stories on this website talking about <platform>'s being garbage over all of its existence.

mayhemducks 1y ago

The Apple calculator IS terrible. When I need a calculator I reach for my HP from high school which still works perfectly even though it only has like 16MB of memory. Kind of sad really.

AznHisoka 1y ago

What data is there that shows software quality is collapsing though? A few anecdotes isnt proof.

bitwize 1y ago

I remember getting into a debate with a professor, who firmly told me human programmers would soon be obsolete because all code would be generated by tools. I told him it wouldn't be very good code and he said, "Who cares? Hardware is cheap!" This was 2002. At the time it was code generation from UML, not AI, but let's face facts: people have failed to give sufficient fucks about software quality for a long time now. We're just at a new low point in a very long, very steep drop; the only significance of this low point is that it means we're still falling. Recently, I saw a video[0] about the game Sonic Racing CrossWorlds. The creator seemed impressed that Sega managed to put out a decently optimized Unreal Engine 5 game, that could hit 60fps without relying on generated-frame fuckery-duckery, and which provided a lot of content in less than 20 GiB. It used to be that optimization to obtain a decent frame rate and avoid bloat was table stakes for game dev. But it's 2025, and this is where we are now. Because video games are enterprise software, the tooling for gamedev itself is optimized around the path of "get the slop out quickly then fix later (maybe)". [0] https://m.youtube.com/shorts/RRToQRmzCDI https://m.youtube.com/shorts/RRToQRmzCDI

imiric 1y ago

The article is spot-on. This should be a direct concern for anyone in the software industry, and given that software is eating the world, to society and governments at large. The key phrase is this: "choose carefully". Software developers have the responsibility to carefully choose their software stacks, the software they depend on, and how their own software is built. And the onus is on people using software to carefully choose software they trust, and software that doesn't restrict their freedoms or exploit their data. Simplicity plays a big role in this choice. The less software we write and depend on, the lower the chances for it to do the wrong thing. We can't guarantee this, of course, given the insane complexity and layers of abstraction we must depend on, but making an effort is all that matters. Sometimes this choice requires trading convenience, time, effort, or money—but the sacrifice is very much worth it.

sciencejerk 1y ago

I would argue that non-safety-critical Software can be sloppy, and continues to be so, because errors do not kill thousands of people. I would imagine the quality bar for safety-critical Software is quite a bit higher...although I do hear about Software quality incidents in the aviation and transportation industries from time to time...

michalsustr 1y ago

The problem is most developers forget a simple thing. Hardware is the platform: https://www.youtube.com/watch?v=rX0ItVEVjHc&t=2891s https://www.youtube.com/watch?v=rX0ItVEVjHc&t=2891s

sha16 1y ago

> Stage 1: Denial (2018-2020) "Memory is cheap, optimization is expensive" I'm not sure if this first premise is right. Optimizing later on makes sense to me and isn't that expensive IMO (this can vary though)

0xWTF 1y ago

I love the idea of high quality code. I do, however, dismiss the energy efficiency issue. > Data centers already consume 200 TWh annually. That's 7.2 x 10^16 joules Some estimates suggest that's low. Let's multiply by 5. Annually, that's 3.6 x 10^17 joules The total radiant power of the sun that hits the Earth's atmosphere is about 174 PW, and about 47% of that lands on the surface. Even with the most drastic global warming models, approximately 100% of that is radiated away at the same time. Evidence: we're not charred embers. So the Earth's power budget, at the surface, is about 81 PW. Annually, that's 2.6 × 10^24 joules Global annual oil production is about 3 x 10^10 barrels, that's 1.8 x 10^20 joules So, the data centers consume 0.2% of the global petroleum budget and 1.4 × 10^-7 of the planet's power budget. The digital economy is about 15% of global GDP, so I submit tech is thoroughly on the right side of their energy conservation balance sheet normalized against other industries and there are other verticals to focus your energy concerns on (e.g. internal combustion engines, including Otto, Brayton, Rankin, and Diesel cycle engines). Among serious people trying to solve serious problems, you would do your credibility a favor by not expending so many words and thus so much reader time on such a miniscule argument. If you want to talk about energy related to code, take the total power consumption of the software engineer's life, and find a way to big-O that. I'm willing to bet they contribute more to global warming driving than all their compute jobs combined.

antisthenes 1y ago
Yes, the moral panic about power consumption from Data centers has always been a red herring at best. Given how cheap renewables are in 2025, and how far we still are from an inverted duck curve world-wide, it's absolute nonsense.

baxuz 1y ago

Talking about "software quality collapse", this blog post with 1 image ships 9mb of JS, 1mb of CSS spread over 14 files, makes 177 requests, takes 4 seconds to load on an M1 MBP, and completely screws up the scrollbar.

jcmontx 1y ago

This seriously reads as "old man yells at cloud". Hardware constraints are not the same as 30 years ago. Processing power and memory keeps getting cheaper. It will be fine

Yoric 1y ago

I absolutely buy that theory. I believe that we the normalization of catastrophe was a natural consequence of VC money: VC don't care about structurally sound companies, in particular structurally sound products, what they want is a unicorn that can produce a good enough prototype and exit at enormous profit. Consequently, VC-backed companies invest in tools that make prototyping easier and in developers who are hopefully good at prototyping (or at least write code quickly), and ignore everything else. And since the surviving VC-backed companies become giants (or at least everybody believes that they will), everybody follows their lead. And of course, LLMs are the next stage of that. I've seen this in various domains. I've seen this with IoT devices coded with the clear (but unsaid) assumption that they will never be upgraded. I've seen this with backends coded with the clear (but unsaid) assumption that the product will have failed before any security hole is exploited. I've seen software tools developed and shipped with the clear (but unsaid) assumption that they're meant to woo the investors, not to help the users. We're going to pay for this, very hard. By doing this, we have turned cyber warfare – something that was a fantasy a few decades ago – into an actual reality. We have multiplied by several orders of magnitude the amount of resources we need to run basic tools. And it's a shame, because there _is_ a path towards using LLMs to produce higher quality code. And I've seen a few teams invest in that, discreetly. But they're not making headlines and it's even possible that they need to be stealthy in their own orgs, because it's not "productive" enough.

jjk166 1y ago

The cause and effect here are reversed. Ever increasing abstraction does not reduce quality, ever reduced emphasis on quality leads to greater abstraction. You only abstract away the nuances when you stop caring about them. When you have so much memory that you can tolerate a massive memory leak, fixing the memory leak gets deprioritized and the tool that saves time at the expense of abstracting away memory management becomes more attractive. The real issue though is that the costs of these abstractions are obfuscated after the fact. There is almost certainly low hanging fruit that could massively improve performance and reduce operating costs with little effort, but there is not a will to analyze and quantify these costs. Adding new features is sexy, it's the stuff people want to work on, it's the stuff people get promoted for doing. You don't get promoted for preventing the issue that never happened, even if it saves billions of dollars. You can't brag about making your software imperceptibly faster, even if it saves a tremendous amount of time across the user base. Software has always only needed to be good enough, and the threshold for good enough has been lowering exponentially with hardware improvements since time immemorial. The system is not set up to favor people who care. If nobody cares, nothing will ever improve. If there were a good way to track every time a fixed bug would have been triggered had it gone unfixed, and if that cost were quantified and visible, there would be a massive push for better quality.

renewiltord 1y ago

Online community that rages against AI slop nonetheless really enjoys AI slop. Hahaha brilliant.

unglaublich 1y ago

Bla bla same story from when C replaced assembly.

deleted 1y ago

[deleted]

jrm4 1y ago

There is no solution here besides "skin-in-the-game." None. The process will be harmful and lazy until someone or something recieves serious consequences for getting it wrong. That's all.

zggf 1y ago

"Instead of addressing fundamental quality issues, Big Tech has chosen the most expensive possible response: throw money at infrastructure." "They're spending 30% of revenue on infrastructure (historically 12.5%). Meanwhile, cloud revenue growth is slowing." Isn't this because: cloud gets more popular -> more big players join -> they compete with each other -> margins get lower?

philodeon 1y ago

I thought one of the benefits of diversity and inclusion in tech was that it would lead to better software quality? When does that kick in?

halis 1y ago

Very good article. I feel that corporate executives as a class are causing this huge mess from a simple lack of understanding of software engineering. Throw more bodies at the problem has never worked in software engineering, in fact that's a surefire way to make things worse and slower. Throwing more bodies + AI at the problem is even worse because the speed at which things get worse is much faster. If you have people that don't understand the implications of what they are doing, you're on the path of heading over a cliff some day. AI is a useful tool, when put into the hands of someone that is competent. They can fire junior devs and refuse to hire them as much as they want, this strategy will come home to roost sooner rather than later. If a C-level executive thinks that AI can replace their engineers, then I say do it now. Go all in and replace them all right now. Use only AI and a few VP-level AI whisperers and let the catastrophic failure that this will result in happen faster so we can get back to reality quicker.

timka 1y ago

Lawrence J. Dickson anticipated this almost 2 decades ago in a research he did for DARPA in 2007[1]. Just look at the software development viscosity graph. [1] http://pros.to/wide.pdf#figure.caption.11 http://pros.to/wide.pdf#figure.caption.11