20 ms·

Trillions spent and big software projects are still failing

• franktankbank 11mo ago
  > Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. Somehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.
  • ruralfam 11mo ago
    My reaction also. 80K payroll rules!!! Without much prompt effort, I got about 350K Canada Federal Service employees (sorry if not correct).
    • dmix 11mo ago
      Sounds like they put zero effort into simplifying those rules the first time around. Now in the new project they put together a committee to attempt it > The main objective of this committee also includes simplifying the pay rules for public servants, in order to reduce the complexity of the development of Phoenix's replacement. This complexity of the current pay rules is a result of "negotiated rules for pay and benefits over 60 years that are specific to each of over 80 occupational groups in the public service." making it difficult to develop a single solution which can handle each occupational groups specific needs.
      • stackskipton 11mo ago
        I have worked on government payroll systems, simplifying those rules is almost impossible from political PoV. They are generally a combo of weird laws, court cases, union contracts and more. Any time you think about touching them, the people who get those salaries come out in droves and no one else cares so government has every incentive to leave them alone.
        • tehjoker 11mo ago
          You could simplify them if you made sure the people getting them got overall more money ;) The government doesn't want to do that though.
          • BeFlatXIII 11mo ago
            The taxpayers might get rather uppity, too.
            • tehjoker 11mo ago
              Do they get uppity about all the wars?
              • BeFlatXIII 11mo ago
                Not enough of them, unfortunately.
      • franktankbank 11mo ago
        Oh great a committee!
        • AndrewDucker 11mo ago
          Committees are how you discover what the problems are and agree solutions. No single person is going to understand all of the history and legality involved, or be able to represent the people on all sides of this mess. Yes, this means discussion, investigation, almost certainly months of effort to find something that works, and lots of compromise. That's how adults deal with complex situations.
  • QuercusMax 11mo ago
    Wasn't the Agile movement kicked off by a group of people writing payroll software for Chrysler? https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens... Payroll systems seem to be a massively complicated beast.
    • franktankbank 11mo ago
      You don't want to get me started on Agile.
    • array_key_first 11mo ago
      Arbitrary payroll is absurdly complicated. The trick is to not make it arbitrary - have a limited amount of stuff you do, and always have backdoors to manually pushing data through payroll.
• ZeroConcerns 11mo ago
  Yup, and with an equal amount of mindblowing-units-of-money spent, infrastructure projects all around me are still failing as well, or at least being modified (read: downsized), delayed and/or budget-inflated beyond recognition. So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones. "Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent! Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?
  • sebastos 11mo ago
    Nailed it, but I fear this wisdom will be easily passed by by someone who doesn’t already intuit it from years of experience. Like the Island de la Muerta: wisdom that can only be found if you already know where it is.
  • graemep 11mo ago
    One of the problems is scale. Large scale systems tend to fail. large centralised and centrally managed systems with big budgets and large numbers of people who need to coordinate, lots of people with an interest in the project pushing and lobbying for different things. Multiple smaller systems is usually a better approach, where possible. Not possible for things like transport infrastructure, but often possible for software.
    • AlexandrB 11mo ago
      > Not possible for things like transport infrastructure It depends what you define as a system. Arguably a lot of transport infrastructure is a bunch of small systems linked with well-understood interfaces (e.g. everyone agrees on the gauge of rail that's going to be installed and the voltage in the wires). Consider how construction works in practice. There are hundreds or thousands of workers working on different parts of the overall project and each of them makes small decisions as part of their work to achieve the goal. For example, the electrical wiring of a single train station is its own self-contained system. It's necessary for the station to work, but it doesn't really depend on how the electrical system is installed in the next station in the line. The electricians installing the wiring make a bunch of tiny decisions about how and where the wires are run that are beyond the ability of someone to specify centrally - but thanks to well known best practices and standards, everything works when hooked up together.
• JohnMakin 11mo ago
  > "Why worry about something that isn’t going to happen?” Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic. It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size. Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.
  • parados 11mo ago
    > and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic. But the rest of your comment reveals nothing novel other than anyone would find after watching James Cameron's movie multiple times. I suggest you go to the original inquiries (congressional in the US, Board of trade in the UK). There is a wealth of subtle lessons there. Hint: Look at the Admiralty Manual of Seamanship that was current at that time and their recommendations when faced with an iceberg. Hint: Look at the Board of Trade (UK) experiments with the turning behaviour of the sister ship. In particular of interest is the engine layout of the Titanic and the attempt by the crew, inexperienced with the ship, to avoid the iceberg. This was critical to the outcome. Hint: Look at the behaviour of Captain Rostron. Lots of lessons there.
• BirAdam 11mo ago
  I study and write quite a bit of tech history. IMHO from what I've learned over the last few years of this hobby, the primary issue is quite simple. While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning. Typically, software folks build new and every generation of software developers must relearn the same problems.
  • tristor 11mo ago
    This is one part of the issue. The other major piece of this that I've seen over more than two decades in industry is that most large projects are started by and run by (but not necessarily the same person) non-technical people who are exercising political power, rather than by technical people who can achieve the desired outcomes. When you put the nexus of power into the hands of non-technical people in a technical endeavor you end up with outcomes that don't match expectations. Larger scale projects deeply suffering from "not knowing what we don't know" at the top.
    • cjbgkagh 11mo ago
      Sometimes giving people what they want can be bad for them; management wants cheap compliant workers, management gets cheap compliant workers, and then the projects fall apart in easily predictable and preventable ways. Because such failures are so common management typically isn’t punished when they do so it’s hard to keep interests inline. And because many producers are run on a cost plus basis there can be a perverse incentive to do a bad job, or at least avoid doing a good one.
    • mbesto 11mo ago
      If this were true all of the time then the fix would be simple - only have technical people in charge. My experience has shown that this (only technical people in charge) doesn't solve the problem.
      • fragmede 11mo ago
        If people didn’t work, maybe we should put an LLM in charge instead.
• mdavid626 11mo ago
  It’s so “nice” to know, that trillions spent on AI not only won’t make this better, but it’ll make it significantly worse.
  • fransje26 11mo ago
    "Worse" won't even start to describe the economical crisis we will be in once the bubble bursts. And although that, in itself, should be scary enough, it is nothing compared to the political tsunami and unrest it will bring in its wake. Most of the Western world is already on shaky political ground, flirting with the extreme-right. The US is even worse, with a pathologically incompetent administration of sociopaths, fully incapable of coming up with the measures necessary to slow down the train of doom careening out of control towards the proverbial cliff of societal collapse. If the societal tensions are already close to breaking point now, in a period of relative economical prosperity, I cannot start to imagine what they will be like once the next financial crash hits. Especially one in the multi trillion of dollars. They say that humanity progresses through episodes of turmoil and crisis. Now that we literally have all the knowledge of the world at our fingertips, maybe it is time to progress past this inadequate primeval advancement mechanism, and to truly enter an enlightened age where progress is made from understanding, instead of crises. Unfortunately, it looks like it's going to take monumental changes to stop the parasites and the sociopaths from making at quick buck at the expense of humanity.
    • BeFlatXIII 11mo ago
      Year zero now. Reset real estate prices due to sudden lack of demand.
  • keeda 11mo ago
    Not really, by most indications AI seems to be an amplifier more than anything else. If you have strong discipline and quality control processes it amplifies your throughput, but if you don't, it amplifies your problems. (E.g. see the DORA 2025 report.) So basically things will still go where they were always going to go, just a lot faster. That's not necessarily a bad thing.
• supportengineer 11mo ago
  The purpose of a system is what it does. 1. Enable grift to cronies 2. Promo-driven culture 3. Resume-oriented software architecture
• mariopt 11mo ago
  > IT projects suffer from enough management hallucinations and delusions without AI adding to them. Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it. It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.
  • tehjoker 11mo ago
    Easy, just imagine a 1GB array as a 2.5mm long square in RAM (assuming a DRAM cell is 10nm). Now it's physical.
  • apercu 11mo ago
    I came to opposite conclusions. Technology is pretty easy, people are hard and the business culture we have fostered in the last 40 years gets in the way of success.
• dmix 11mo ago
  So in the 1990s Canada failed to do a payroll system where they paid Accenture Canada $70M Then in 2010s they spent $185M on a customized version of IBM's PeopleSoft that was managed directly by a government agency https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system And now in 2020s they are going to spend $385M integrating an existing SaaS made by https://en.wikipedia.org/wiki/Dayforce https://en.wikipedia.org/wiki/Dayforce That's probably one of the worst and longest software failures in history.
  • bryanlarsen 11mo ago
    Oh, it's much more interesting than that. Phoenix started as an attempt to create a gun registry. Ottawa had a bunch of civil servants that'd be reasonably compotent at overseeing such a thing, but the government decided that it wanted to build it in Miramichi, New Brunswick. The relevant people refused to move to Miramichi, so the project was built using IBM contractors and newbies. The resulting fiasco was highly predictable. Then when Harper came in he killed the registry mostly for ideological reasons. But then he didn't want to destroy a bunch of jobs in Miramichi, so he gave them another project to turn into a fiasco.
    • stackedinserter 11mo ago
      > Then when Harper came in he killed the registry mostly for ideological reasons. The registry was started mostly for ideological reasons.
      • bryanlarsen 11mo ago
        Sure, that's obvious. What isn't obvious is why it was killed. It could have been killed because it was a giant cluster@#$%, that would have been reason enough. But if it would have been killed for that reason, they wouldn't have let the cluster%$## spread.
  • robocat 11mo ago
    > Canada failed to do a payroll system New Zealand tried to do a new payroll system for teachers called Novopay which imploded spectacularly and is still creating grief. The system is now called EdPay (the government decided to take over the privately created system). The total cost of the debacle was north of $200M NZD. Somehow they managed to fail to replace a working system!
• bigbuppo 11mo ago
  As someone that has seen technological solutions applied when they make no sense, I think the next revolution in business processes will be de-computerization. The trend has probably already started thank to one of the major cloud outages.
  • stock_toaster 11mo ago
    > de-computerization I would think cloud-disconnectedness (eg. computers without cloud hosted services) would come far before de-computerization.
  • senshan 11mo ago
    Can you please provide a few examples to get the gist of such trend? Honest question.
• add-sub-mul-div 11mo ago
  An endless succession of new tools, methodologies, and roles but failure persists because success is rooted in good judgment, wisdom, and common sense.
• jmyeet 11mo ago
  This has dot-com bubble written all over it. But there are some deeper issues. First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error. Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat. Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave. Who will buy the results of all this AI if there’s no one left with a job? Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices. We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.
  • ctoth 11mo ago
    Systemically, how would you solve homelessness, if I gave you a trillion dollars?
    • jddj 11mo ago
      A trillion in a money market fund @ 5% is 50B/year. Over the course of a few years (so as to not drive up the price of politicians too quickly) one could buy the top N politicians from most countries. From there on out your options are many. After a decade or so you can probably have your trillion back.
      • ctoth 11mo ago
        I really do like this answer, but it would seem to have the property of being anti-inductive in that the cost for current politicians is so low because nobody is doing it at scale but if someone did that would force other people to ... well, it's an interesting thought experiment at least!
  • tonyedgecombe 11mo ago
    The article isn't really about AI (for a change).
• ThaDood 11mo ago
  So, I'm not a dev nor a project manager but I found this article very enlightening. At the risk of asking a stupid question and getting a RTFM or a LMGTFY can anyone provide any simple and practical examples of software successes at a big scale. I work at a hospital so healthcare specific would be ideal but I'll take anything. FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh. My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.
  • BenoitEssiambre 11mo ago
    Unix and Linux would be your quintessential examples. Unix was an effort to take Multics, an operating system that had gotten too modular, and integrate the good parts into a more unified whole (book recommendation: https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/dp/1695978552/ https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/d...). Even though there were some benefits to the modularity of Multics (apparently you could unload and replace hardware in Multics servers without reboot, which was unheard of at the time), it was also its downfall. Multics was eventually deemed over-engineered and too difficult to work with. It couldn't evolve fast enough with the changing technological landscape. Bell Labs' conclusion after the project was shelved was that OSs were too costly and too difficult to design. They told engineers that no one should work on OSs. Ken Thompson wanted a modern OS so he disregarded these instructions. He used some of the expertise he gained while working on Multics and wrote Unix for himself (in three weeks, in assembly). People started looking over Thompson's shoulder being like "Hey what OS are you using there, can I get a copy?" and the rest is history. Brian Kernighan described Unix as "one of" whatever Multics was "multiple of". Linux eventually adopted a similar architecture. More here: https://benoitessiambre.com/integration.html https://benoitessiambre.com/integration.html
    • prmph 11mo ago
      Are you equating success with adoption or use? I would say there are lot's of software that are widely used but are a mess. What would be a competitor to linux that is also FOSS? If there's none, how do you assess the success or otherwise of Linux? Assume Linux did not succeed but was adopted, how would that scenario look like? Is the current situation with it different from that?
• exabrial 11mo ago
  The biggest reason is developer ego. Devs see their code as artwork an extension of themselves, so it's really hard to have critical conversations about small things and they erupt into holy wars. Off hand: * Formatting * Style * Conventions * Patterns * Using the latest frameworks or whats en-vogue I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.
  • ctoth 11mo ago
    The UK Post Office lied and made people kill themselves ... because of dev ego?
    • bzzzt 11mo ago
      To me it screams more like an organization not wanting to assume blame and risk paying for their errors.
  • AlotOfReading 11mo ago
    As far as I can see in the modern tech industry landscape, virtually everyone has adopted style guides and automatic formatting/linting. Modern languages like Go even bake those decisions into the language itself. I'd consider managing that stuff essentially table-stakes in big orgs these days. It doesn't stop projects from failing in highly expensive and visible ways.
    • bossyTeacher 11mo ago
      > in the modern tech industry landscape, virtually everyone has adopted style guides and automatic formatting/linting the modern tech industry landscape is in absolute terms is small compared to the wider tech industry landscape afaik
  • exabrial 11mo ago
    Ironically, the downvotes pretty much prove this is exactly correct.
• apercu 11mo ago
  Hot take: It's not technical problems causing these projects to fail. It's leadership and accountability (well, the lack of them).
  • AnimalMuppet 11mo ago
    And that often takes a particular form: The requirements never converge, or at least never converge on anything realistically buildable.
• 0xbadcafebee 11mo ago
  Software projects fail because humans fail. Humans are the drivers of everything in our world. All government, business, culture, etc... it's all just humans. You can have a perfect "process" or "tool" to do a thing, but if the human using it sucks, the result will suck. This means that the people involved are what determines if the thing will succeed or fail. So you have to have the best people, with the best motivations, to have a chance for success. The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences. Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact) That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.
  • beezlebroxxxxxx 11mo ago
    If software engineers want to be referred to as "engineers" then they should actually learn about engineering failures. The industry and educational pipeline (formal and informal) as a whole is far more invested in butterfly chasing. It's immature in the sense that many people with decades of experience are unwilling to adopt many proven practices in large scale engineering projects because they "get in the way" and because they hold them accountable.
    • QuiDortDine 11mo ago
      Surely you mean managers, right? Most developers I interact with would love to do things the right way, but there's just no time, we have to chase this week's priority!
• John23832 11mo ago
  I often see big money put behind software projects, but the money then makes stake holders feel entitled to get in the way.
• parasubvert 11mo ago
  Working on AI that helps to manage IT shops that learns from failure & success might be better for both results and culture than most IT management roles, a profession (painting an absurdly broad brush) that tends to attract a lot of miserable creatures.
  • gishh 11mo ago
    ... If this happens, the next hacks will be context poisoning. A whole cottage industry will pop around preserving and restoring context. Sounds miserable. Also, LLMs don't learn. :)
    • parasubvert 11mo ago
      LLMs themselves don’t learn but AI systems based around LLMs can absolutely learn! Not on their own but as part of a broader system: RLHF leveraging LoRAs that get re-incorporated as model fine tunings regularly, natural language processing for context aggregation, creative use of context retrieval with embeddings databases updated in real time, etc.
• nacozarina 11mo ago
  managing software requirements and the corresponding changes to user/group/process behaviors is by far the hardest part of software development, and it is a task no one knows how to scale. absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out. every time, ugly baby
• ChrisMarshallNY 11mo ago
  > While hardware folks study and learn from the successes and failures of past hardware, software folks do not. I guess that’s the real problem I have with SV’s endemic ageism. I was personally offended, when I encountered it, myself, but that’s long past. I just find it offensive, that experience is ignored, or even shunned. I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.
  • pork98 11mo ago
    Why do you find it offensive? It’s not personal. Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right? Even evolution shuns experience, all but throwing most of it out each generation, with a scant few species as exceptions.
    • pkilgore 11mo ago
      I think you're mistaking the funding and starting of companies with the execution of their vision through software engineering -- the entire point of the article, and the OP.
    • Bjartr 11mo ago
      > Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right? Not at all. The mistake to learn from in Webvan's case was expanding too quickly and investing in expensive infrastructure all before achieving product-market fit. Not that they delivered groceries.
    • antonvs 11mo ago
      This is a classic straw man argument, which depends on the assumption that all people of a certain age would think a certain way. Also, your understanding of evolution is incorrect. All species on Earth are the results of an enormous amount of accumulated "experience", over periods of up to billions of years. Even the bacteria we have today took hundreds of millions of years to reach anything similar to their current form.
    • 11mo ago
• x0x0 11mo ago
  The article is kind of dumb. eg it hangs its hat on the Phoenix payroll system, which > Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data. So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees! IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects. Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple. [1] https://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01_e_42666.html https://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_0...
• MattRogish 11mo ago
  The lesson from “big software projects are still failing” isn’t that we need better methodologies, better project management, or stricter controls. The lesson is "don't do big software projects". Software is not the same as building in the physical world where we get economies of scale. Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.) Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software. Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale. Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process. Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared. It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself. (Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally). We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix. The lesson is: "never build big software". (Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)
• lawlessone 11mo ago
  Every improvement will be moderated increased demands from management, crunch, pressure to release, "good enough", add this extra library that monetizes/spys on the customer etc In the same way that hardware improvements are quickly gobbled up by more demanding software. The people doing the programming will also be more removed technically. I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.
  • lawlessone 11mo ago
    will be moderated by* increased demands.
• neilv 11mo ago
  On some of the infamous large public IT project failures, you just have to look at who gets the contract, how they work, and what their incentives are. (For example, don't hire management consulting partner smooth talkers, and their fleet of low-skilled seat-warmers, to do performative hours billing.) It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology). But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs. Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.
  • JBlue42 11mo ago
    > Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology). Was there any literature or other findings that you came across that ended up clicking and working for you that you can recommend to us?
    • neilv 11mo ago
      I could blather for hours around this space. A few random highlights: * The very first thing I read about requirements was Weinberg, and it's still worth reading. (Even if you are a contracting house, with a hopeless client, and you want to go full reactive scrum participatory design, to unblock you for sprints with big blocks of billable hours, not caring how much unnecessary work you do... at least you will know what you're not doing.) * When interviewing people about business or technical, learn to use a Data Flow Diagram. You can make it accessible to almost everyone, as you talk through it, and answer all sorts of questions, at a variety of levels. There are a bunch of other system modeling tools you can use, at times, but do not underestimate the usefulness and accessibility of a good DFD. * If you can (or have to) plan at all, find and learn to use a serious Gantt chart centric planning tool (work breakdown, dependencies, resource allocations, milestones), and keep it up to date (which probably includes having it linked with whatever task-tracking tool you use, but you'll usually also be changing it for bigger-picture reasons too). Even if you are a hardware company, with some hard external-dependency milestones, you will be changing things around those unmoveables. And have everyone work from the same source of truth (everyone can see the same Gantt chart and the task * Also learn some kind of Kanban-ish board for tasking, and have it be an alternative view on the same data that's behind the Gantt view and the tasks/issues that people can/should/are working on at the moment, and anything immediately getting blocked. * In a rare disruptive startup emergency, know when to put aside Gantt, and fall back to an ad hoc text file or spreadsheet of chaos-handling prioritization that's changing multiple times per day. (But don't say that your startup is always in emergency mode and you can never plan anything, because usually there is time for a Kanban board, and usually you should all share an understanding of how those tasks fit into a larger plan, and trace back to your goals, even if it's exploratory or reactive.) * Culture of communicating and documenting, in low-friction, high-value, accessible ways. Respect it as team-oriented and professional * Avoid routine meetings; make it easy to get timely answers and discussion, as soon as possible. This includes reconsidering how accessible upper leadership should be: can you get closer to being responsive to the needs of the work on the project (e.g., if anyone needs a decision from the director/VP/etc., then quickly prep and ask, maybe with an async message, but don't wait for weekly status meeting or to schedule time on their calendar). * Avoid unnecessary process. Avoid performances. * People need blocks of time when they can get flow. Sometimes for plowing through a big chunk of stuff that only requires basic competence, and sometimes when harder thinking is required. * Be very careful with individual performance metrics. Ideally you can incentive everyone to be aligned towards team success, through monetary incentives (e.g., real equity for which they can affect the value) and through culture (everyone around you seems to work as a team, and you like that, and that inspires you). I would even start by asking if we can compensate everyone equally, shun titles, etc., and how close can we practically get to that. * Be honest about resume-driven-development. It doesn't have to be a secret misalignment. Don't let it be motivated solely as a secret goal of job-hoppers that is then lied about, or it will probably be to the detriment of your company (and also, that person will job-hop, fleeing the mess they made). If you're going to use new resume keyword framework for a project, the whole team should be honest that, say, there's elements of wanting to potentially get some win from it, wanting to trial it for possible greater use and build up organizational expertise in it, and also that it's a very conscious and honest perk for the workers to get to use the new toy. * Infosec is an unholy dumpster fire, throughout almost the entire field. Decide if you want to do better, and if so, then back it up with real changes, not CYA theatre and what someone is trying to sell you. * LeetCode frat pledging interviews select for so much misaligned thinking, and also signals that you are probably just more of the same as the low bar of our field, and people shouldn't take you seriously when you try to tell them you want to do things better. * Nothing will work well if people aren't aligned and honest.
• amai 11mo ago
  To stop failing we could use AI to replace managers not software developers.
  • an0malous 11mo ago
    No need to waste GPUs, a simple bash script that alternates between asking for status updates and randomly changing requirements would do
    • mring33621 11mo ago
      We are seeking improvements, not the status quo.
• oldandboring 11mo ago
  Almost nobody who works in software development is a licensed professional engineer. Many are even self-taught, and that includes both ICs and managers. I'm not saying this is direct causation but I do think it odd that we are so utterly dependent on software for so many critical things and yet we basically YOLO its development compared to what we expect of the people who design our bridges, our chemicals, our airplanes, etc.
  • keeda 11mo ago
    Licensing and the perceived rigor it signifies is irrelevant to whether something can be considered "professional engineering." Engineering exists at the intersection of applied science, business and economics. So most software projects can be YOLO'd simply because the economics permit it, but there are others where the high costs necessitate more rigor. For instance, software in safety-critical systems is highly rigorously developed. However that level of investment does not make sense for run-of-the-mill internal LOB CRUD apps which constitute the vast majority of the dark matter of the software universe. Software engineering is also nothing special when it comes to various failure modes, because you'll find similar examples in other engineering disciplines. I commented about this at length a few days ago: https://news.ycombinator.com/item?id=45849304 https://news.ycombinator.com/item?id=45849304
• serial_dev 11mo ago
  This is what I’ve been thinking about when I talk to other people in software development when they can’t stop talking about how efficient they are with AI… yet they didn’t ship anything in their startup, or side project, or in a corporate setting, the project is still bug riddled, the performance is poor, now there code quality suffers too as people barely read what Cursor (etc) are spitting out. I have “magical moments” with these tools, sometimes they solve bugs and implement features in 5 minutes that I couldn’t do in a day… at the same time, quite often they are completely useless and cause you to waste time explaining things that you could probably just code yourself much faster.
• shevy-java 11mo ago
  I spent way less - and they still fail!
• runningmike 11mo ago
  There is no such thing as ‘simplicity science’ that can be directly applied when dealing with IT problems. However, many insights of complexity science are applicable to solving real world IT problems. People love simple solutions. However Simple is a scam, https://nocomplexity.com/simple-is-a-scam/ https://nocomplexity.com/simple-is-a-scam/ There are no generic, simple solutions for complex IT challenges. But there are ground rules for finding and implementing simple solutions. I have created a playbook to prevent IT diasasters, The art and science towards simpler IT solutions see https://nocomplexity.com/documents/reports/SimplifyIT.pdf https://nocomplexity.com/documents/reports/SimplifyIT.pdf
• 827a 11mo ago
  Slightly related but unpopular opinion I have: I think software, broadly, today is the highest quality its ever been. People love to hate on some specific issues concerning how the Windows file explorer takes 900ms to open instead of 150ms, or how sometimes an iOS 26 liquid glass animation is a bit janky... we're complaining about so much minutia instead of seeing the whole forest. I trust my phone to work so much that it is now the single, non-redundant source for keys to my apartment, keys to my car, and payment method. Phones could only even hope to do all of these things as of like ~4 years ago, and only as of ~this year do I feel confident enough to not even carry redundancies. My phone has never breached that trust so critically that I feel I need to. Of course, this article talks about new software projects. And I think the truth and reason of the matter lies in this asymmetry: Android/iOS are not new. Giving an engineering team agency and a well-defined mandate that spans a long period of time oftentimes produces fantastic software. If that mandate often changes; or if it is unclear in the first place; or if there are middlemen stakeholders involved; you run the risk of things turning sideways. The failure of large software systems is, rarely, an engineering problem. But, of course, it sometimes is. It took us ~30-40 years of abstraction/foundation building to get to the pretty darn good software we have today. It'll take another 30-40 years to add one or two more nines of reliability. And that's ok; I think we're trending in the right direction, and we're learning. Unless we start getting AI involved; then it might take 50-60 years :)
• mschuster91 11mo ago
  No big surprise. Taking a shitty process and "digitalizing" it will lead to a shitty process just on computers in the best case, in the worst case everything collapses.
• SatvikBeri 11mo ago
  Do non-software projects succeed at a higher rate in any industry? I get the impression that projects everywhere go over time, over budget, and frequently get canceled.
  • venturecruelty 11mo ago
    How many bridges have you used that have collapsed? How much software have you used that has been broken or otherwise not served your interests? If we built the rest of society like we build software, millions of people would be dead.
    • agos 11mo ago
      the UK Post Office scandal would be the equivalent of the Morandi bridge collapsing - the big, catastrophic failure you hope to see few times in your lifetime. but bridges collapsing is not the only failure mode for non software projects. I know plenty of newly built houses that had serious issues with insulation, wiring, painting, heating infrastructure, etc.
    • nitwit005 11mo ago
      The reason bridges don't fail often is because they over-build them. There's no obvious equivalent with software. One mistake in a large code base can make it fail. Bridges to go ridiculously over budget and schedule all the time, however.
• rossdavidh 11mo ago
  It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features. If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. There is no software development process which reliably produces software that works at scale without doing it small, and medium sized, first, and fixing what goes wrong before you go big.
  • solatic 11mo ago
    That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen. Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.
    • dustingetz 11mo ago
      we get paid to add to it, we don’t get paid to take away
      • cjfd 11mo ago
        Now there is your problem. It is only true in the context of grave incompetence, though. I have worked on tickets with 'remove' in the title.
    • Jtsummers 11mo ago
• AtlasBarfed 11mo ago
  Software was failing and mismanaged. So we added a language and cultural barrier, 12 hour offset, and thousands of miles of separation with outsourcing. Software was failing and mismanaged. So now we will take the above failures, and now tack on an AI "prompt engineering" barrier (done by the above outsourced labor). And on top of that, all engineers that know what they are doing are devalued from the market, all the newer engineers will be AI braindead. Everything will be fixed!
• darepublic 11mo ago
  Is it a failure if we ship the project a year late? What if everyone involved would have predicted exactly that outcome
• satisfice 11mo ago
  Systematic decimation of test teams, elimination of test managers, and contemptuous treatment of the role of tester over the past 40 years has not yet led to a more responsible software industry. But maybe if we started burning testers at the stake all these problems will go away?
  • icedchai 11mo ago
    Many specialties were eliminated / absorbed over the past few decades. I started working almost 30 years ago. Today, I rarely see dedicated testers, just like I rarely see dedicated DBAs. Sysadmins went away with the "DevOps" movement. Now they are cloud engineers who are more likely to understand a vendor-specific implementation than networking fundamentals.
    • satisfice 11mo ago
      Except testers are needed. Testing is not merely a technical role. It's a social role. It's a commitment to think differently from the rest of the team. We do this because that provides insurance against complacency. But, by the nature of testing, we testers are outsiders. No one is fully comfortable with a tester in the room, unless they are more afraid of failure than irritation.
• zaptheimpaler 11mo ago
  This should be a criticism of the kinds of bloated firms that take on large government projects, the kinds of people they hire, the incentives at play, the bidding processes, the corruption and all the rest. It has very little to do with software and more just organizations that don't face any pressure to deliver.
• user3939382 11mo ago
  How much money do you need to build a skyscraper on top of a tarpit? None because it’s not possible. The whole stack has to be gutted. I can do it but no one wants to listen so I’ll do it myself.
• sandeepkd 11mo ago
  A slightly different take, its probably more of people failure, the lack of required expertise, skillset, motivation and coordination. People have motivations to do the job to make a living, success of any long term project is rarely the driving factor for most people working on it. People would know ahead of time when a project is going towards the direction of failure, its just how the things are structured. From systems perspective, an unknown system/requirement would be a good example where you build iteratively, a known set of requirements should give good enough idea about the feasibility and rough timelines even if its complex.
• smithkl42 11mo ago
  Plausible article, but it reads like a preschooler frustrated that his new toy is broken. "Fix it! Make it work!" - without ever specifying how. Granted, this is an exceedingly hard problem, and I suppose there's some value in reminding ourselves of it - but I'd much rather read thoughts on how to do it better, not just complaints that we're doing it poorly.
• locallost 11mo ago
  Worth a view also. Is software engineering still an oxymoron? https://youtu.be/D43PlUr1x_E?si=em2nNYuI8WDvtP21 https://youtu.be/D43PlUr1x_E?si=em2nNYuI8WDvtP21
• watersb 11mo ago
  It's possible that most business projects fail. Most advertising campaigns fail.
• csours 11mo ago
  There are 2 big problems with large software projects: 1. Connecting pay to work - estimates (replanning is learning, not failure) 2. Connecting work to pay - management (the world is fractal-like, scar tissue and band-aids) I do not pre-suppose that there are definite solutions to these problems - there may be solutions, but getting there may require going far out of our way. As the old farmer said "Oh, I can tell you how to get there, but if I was you, I wouldn't start from here" 1. Pay to Work - someone is paying for the software project, and they need to know how much it will cost. Thus estimates are asked for, an architecture is asked for, and the architecture is tied to the estimates. This is 'The Plan!'. The project administrators will pick some lifecycle paradigm to tie the architecture to the cost estimate. The implementation team will learn as they do their work. This learning is often viewed as failure, as the team will try things that don't work. The implementation team will learn that the architecture needs to change in some large ways and many small ways. The smallest changes are absorbed in regular work. Medium and Large changes will require more time (thus money); This request for more money will be viewed as a failure in estimation and not as learning. 2. Work to Pay - as the architecture is implemented, development tasks are completed. The Money People want Numbers, because Money People understand how they feel about Numbers. Also these Numbers will talk to other Numbers outside the company. Important Numbers with names like Share Price. Thus many layers of management are chartered and instituted. The lowest layer of management is the self-managed software developer. The software developer will complete development tasks related to the architecture, tied to the plan, attached to the money (and the spreadsheets grew all around, all around [0]). When the developer communicates about work, the Management Chain cares to hear about Numbers, but sometimes they must also involve themselves in failures. It is bad to fail, especially repeated failures at the same kind of task. So managers institute rules to prevent failures. These rules are put in a virtual cabinet, or bureau. Thus we have Rules of the Bureau or Bureaucracy. These rules are not morally bad or good; not factually incorrect or correct, but whenever we notice them, they feel bad; We notice the ones that feel bad TO US. We are often in favor of rules that feel bad to SOMEONE ELSE. You are free to opt out of this system, but there is a price to doing so. ---- Too much writing, I desist from decoding verbiage: Thus it is OK for individuals to learn many small things, but it is a failure for the organization to learn large things. Trying to avoid and prevent failure is viewed as admirable; trying to avoid learning is self-defeating. ---- 0. https://www.google.com/search?q=the+green+grass+grew+all+around https://www.google.com/search?q=the+green+grass+grew+all+aro... > git commit -am "decomposing recapitulating and recontextualizing software development bureaucracy" && git push
• vb-8448 11mo ago
  The main problem are incentives and risks: in most of the cases you are not incentivized to build secure and reliability SW because, most of the time, it's easy to fix it. With particular categories of SW(eg. one distributed on remote system, medical sw, military sw) or HW it's the opposite: a failure it's not so easy to fix so you are incentivized to do a better job. The second problem are big con.
• skywhopper 11mo ago
  Because you don’t just rewrite all your payroll systems with hundreds of variations in one go. That will never work. But they keep trying it. You update the system for one small piece, while reconciling with the larger system. Then replace other pieces over time, broadening your scope until you have improved the entire system. There is no other way to succeed without massive pain.
• semiinfinitely 11mo ago
  > We are left with only a professional and personal obligation to reemphasize the obvious: Ask what you do know, what you should know, and how big the gap is between them before embarking on creating an IT system. If no one else has ever successfully built your system with the schedule, budget, and functionality you asked for, please explain why your organization thinks it can translation: "leave it to us professionals". Gate-keeping of this kind is exactly how computer science (the one remaining technical discipline still making reliable progress) could become like all of the other anemic, cursed fields of engineering. people thinking "hey im pretty sure I could make a better version of this" and then actually doing it is exactly how progress happens. I hope nobody reads this article and takes it seriously
• dvrp 11mo ago
  https://en.wikipedia.org/wiki/Productivity_paradox https://en.wikipedia.org/wiki/Productivity_paradox
• twidledee 11mo ago
  The difference between success and failure of large projects comes down to technical leadership. I've seen it time and time again. Projects that are managed by external consulting companies (name brand or otherwise) have a very poor track record of delivering. An in-house technical lead that is committed to the success of the project will always do better. And yes, this technical lead must have the authority to limit the scope of the system rewrite. Endless scope creep is a recipe for failure. Outside consulting firms will never say "No" to any new request - it means more business for them - their goals are not aligned with the client.
• namegulf 11mo ago
  Throwing money at a problem never works and never will!
• dockd 11mo ago
  If it makes anyone feel better, it's not just software: https://en.wikipedia.org/wiki/Auburn_Dam https://en.wikipedia.org/wiki/Auburn_Dam https://en.wikipedia.org/wiki/Columbia_River_Crossing https://en.wikipedia.org/wiki/Columbia_River_Crossing If you're 97% over budget, are you successful? https://en.wikipedia.org/wiki/Big_Dig https://en.wikipedia.org/wiki/Big_Dig
  • mpyne 11mo ago
    > If you're 97% over budget, are you successful? I don't like this as a metric of success, because who came up with the budget in the first place? If they did a good job and you're still 97% over then sure, not successful. But if the initial budget was a dream with no basis in reality then 97% over budget may simply have been "the cost of doing business". It's easier to say what the budget could be when you're doing something that has already been done a dozen times (as skyscraper construction used to be for New York City). It's harder when the effort is novel, as is often the case for software projects since even "do an ERP project for this organization" can be wildly different in terms of requirements and constraints. That's why the other comment about big projects ideally being evolutions of small projects is so important. It's nearly impossible to accurately forecast a budget for something where even the basic user needs aren't yet understood, so the best way to bound the amount of budget/cost mismatch is to bound the size of the initial effort.
    • dockd 11mo ago
      I just picked one metric from Wikipedia. It was also 22 years late.
      • jgeada 11mo ago
        The Big Dig has been an enormous success though, it has completely revitalized Boston.
      • lmm 11mo ago
        Same question for that though. Was it late because the implementation was done badly or because the original estimate was unrealistic?
• senshan 11mo ago
  In the book "How Big Things Get Done" [1], Bent Flyvbjerg, among other things, identifies one common feature of the projects that do not have large outliers to go over-budget and under-deliver: modularity. Ideally, fractal modularity. His favorite examples: solar power, electric transmission, pipelines, roads. Ironically, IT/software is only slightly better than nuclear power and Olympic games [2]. [1] https://www.amazon.com/-/en/dp/B0B63ZG71H https://www.amazon.com/-/en/dp/B0B63ZG71H [2] https://www.scribd.com/document/826859800/How-Big-Things-Get-Done-The-Surprising-Factors-That-Determine-the-Fate-of-Every-Project-from-Home-Renovations-to-Space-Exploration-and-Everything-in https://www.scribd.com/document/826859800/How-Big-Things-Get...
• scotty79 11mo ago
  People concerned about small benefits from AI should consider that IT and the internet is failing upwards for more than three decades now.
• mring33621 11mo ago
  'Managers' aren't really getting any better as time goes on...
• tedggh 11mo ago
  Looking at other domains where companies are developing complex products in highly regulated industries there’s one thing they all share in common, they invest a lot of capital in infrastructure for testing their designs. I spent years at a company trying to convince upper management in setting up a lab where we could simulate a production environment that would allow us to do a real integration test. It’s an idea hard to sell because testing is actually part of the budget in every project, so lack of testing couldn’t be attributed to our high rate of failures (going over budget fixing bugs during commissioning). Perhaps we should stop calling unit testing, testing so that we don’t confuse people. Until you we don’t put all the pieces together and do a proper stress test under close-to-realistic production conditions, our software cannot be considered tested. I think that’s the case for 99% of software companies.
• venturecruelty 11mo ago
  People are _so amazingly close_ to realizing what is wrong with this entire industry. So close.
  • stackedinserter 11mo ago
    Enlighten us.
• quantum_state 11mo ago
  Interesting article ... with the wisdom of software engineering being forgotten, it will unfortunately get worse ...
• interstice 11mo ago
  Completely off topic but when fonts are the size they are in this article I can't read it, the words don't register as words above a certain size. I assume this isn't normal or it wouldn't be so common...
• nakamoto_damacy 11mo ago
  because most people are incompetent, produce incidental complexity to satisfy internal urge for busy work, and under-think the problem, greatly... that's why, and don't get me started on the morons who run the show
• chickensong 11mo ago
  What a joke blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features as fast as they can. "We take security and reliability very seriously", until management gets wind of the price tag. Guess what areas always get considered last and cut first. We all know. But sure, blame the community at large, not the execs who make bad decisions in the name of short-term profits, then fail upward with golden parachutes into their next gig. And definitely don't blame government for not punishing egregious behavior of corporations. Don't worry, you get a year of free credit monitoring from some scummy company who's also selling your data. Oh and justice was served to the offending corp, who got a one-time fine of $300k, when they make billions every quarter. Maybe if we just outsource everything to AI, consultants, and offshore sweat shops things will improve! Okay cool, good article.
  • tjwebbnorfolk 11mo ago
    > blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features IT != software engineering. IT is a business function that manages a company's internal information. Software engineering is a time-tested process of building software. A lot of projects fail because management thinks that IT is a software engineering department. It is not. It never was, and it never will be. Its incentives will never be aligned such that software engineering projects are set up for success. The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning.
    • chickensong 11mo ago
      I understand, but also, IT is an umbrella term for a wider industry that includes your definition of IT, software, and anything adjacent. If you read the article, you'll see it's the latter being referenced, and why I chose that terminology. > The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning. That's a pretty strong statement. Isn't that the opposite of why the devops movement started?
• jgeada 11mo ago
  Fundamentally this is not a statement about programming or software. It is a statement that management at almost all companies is abysmally inept and are hardly ever held to account. Most sizeable software projects require understanding, in detail, what is needed by the business, what is essential and what is not, and whether any of that is changing over the lifetime of the project. I don't think I've ever been on a project where any of that was known, it was all guess work.
  • scuff3d 11mo ago
    Management is always a huge problem, but software engineers left to their own devices can be just as bad. I very rarely hear actual technical reasons for why a decision was made. They're almost always invented after the fact to retroactive justify some tool or design pattern the developer wanted to use. Capabilities and features get tacked on just because it's something someone wanted to do, not because they solve an actual problem or can be traced back to requirements in any meaningful way. Frankly as an industry we could learn a lot from other engineering fields, aerospace and electrical engineering in particular. They aren't perfect, but in general they're much better at keeping technical decisions tied to requirements. Their processes tend to be too slow for our industry of course, but that doesn't mean there aren't lessons to be learned.
    • N_Lens 11mo ago
      Post fact justification seems to be a 'feature' of most people's cognitive function, according to the latest research. "The mind is just a bullshit maker".
      • OhMeadhbh 11mo ago
        You say "the mind is just a bullshit maker" like it's a bad thing.
  • misja111 11mo ago
    Exactly this. Not just large software projects tend to fail often; also large architectural and infrastructure projects do. There are loads of examples, one famous one for instance is the Berlin Airport. Management is bad at managing large projects. Whatever those projects are. In particular when third parties are involved that have a financial interest.
• npalli 11mo ago
  Kind of strange take as though unique to software. Every sector that is large has issues since ambitious projects stretch what can be done by the current management and organizational practices. All software articles like these hark back to some mythical world smaller in scope/ambition/requirements. Humanity moves forward * Construction and Engineering -- Massive cost overruns and schedule delays on large infrastructure projects (e.g., public transit systems, bridges) * Military and Government -- Defense acquisition programs notorious for massive cost increases and years-long delays, where complex requirements and bureaucratic processes create an environment ripe for failure. * Healthcare -- Hospital system implementations or large research projects that exceed budgets and fail to deliver intended efficiencies, often due to resistance to change and poor executive oversight.
• sanp 11mo ago
  AI will fix this
• jillesvangurp 11mo ago
  Most of the examples here are big government IT projects. But it's unfair to single out software projects here. There are a lot of big government projects that fail or face long and expensive delays. A lot of public sector spending is like that. In fact, you'd be hard pressed to find examples where everything worked on time and on budget. Mostly the issues are non technical and grounded in a lack of accountability and being too big to fail. A lot of these failures are failing top down. Unrealistic expectations, hand wavy leadership, and then that gets translated into action. Once these big projects get going and are burning big budgets and it's obvious that they aren't working, people get very creative at finding ways to tap into these budgets. Here in Germany, the airport in Berlin was opened only a few years ago after being stuck in limbo a decade after it was supposed to open and the opening was cancelled only 2 weeks before it was supposed to happen. It was hilarious, they had signs all over town announcing how they were going to shut down the highway so the interior of the old airport could be transported to the new one. I kid you not. They were going to move all the check-in counters and other stuff over and then bang on it for a day or two and then open the airport. Politicians, project leadership, etc. kept insisting it was all fine right up until the moment they could not possibly ignore the fact that there was lots wrong with the airport and that it wasn't going to open. It then took a decade to fix all that. There's a railway station in Stuttgart that is at this point very late in opening. Nuclear plant projects tend to be very late and over budget too. Government IT projects aren't that different than these. It's a very similar dynamic. Big budgets, decision making is highly political, a lack of accountability, lots of top down pretending it's going to be fine, big budgets and companies looking to tap into those, and a lot of wishful thinking. These are all common ingredients in big project failures. The software methodology is the least of the challenges these projects face.
  • mcny 11mo ago
    It is not just government. Private companies also have the same problem. One reason why aws got so big is because it took months to get infrastructure to provision a virtual machine.
    • hackernewds 11mo ago
      something left out there with government though is incentive misaligned and hence corruption, which is smaller in a private scale (exists nonetheless)
• rester324 11mo ago
  It's good that the author makes the distinction between developers and managers. This distinction is rarely made and most media outlets talk about the wrongdoings of developers, who are almost never the decision makers of failing projects. It's quite the opposite, they are the ones who if brave enough criticize bad management practices and the lack of proper management of the software project.
• ropable 11mo ago
  I'm pretty sure that we can remove the word "software" from the article headline and it remains just as true. I don't believe that software projects are unique in this regard: big, complex projects are big and complex, and prone to unexpected issues, scope creep, etc. Throw in multiple stakeholders, ineffective management, the sunk cost fallacy etc. and it's a wonder that any large projects get finished at all.
• aryehof 11mo ago
  Failure typically comes from two directions. Unknown and changing requirements, and management that relies on (often external) technical (engineering) leadership that is too often incompetent. These projects are often characterized by very complex functional requirements, yet are undertaken by those who primarily only know (and endlessly argue about) non-functional requirements.
• rester324 11mo ago
  I like that the author propagates software developer liability. That makes sense. Unless we introduce such system, the incentives are not there to avoid failure https://queue.acm.org/detail.cfm?id=3489045 https://queue.acm.org/detail.cfm?id=3489045 https://therecord.media/cybersecurity-software-liability-standards-white-house-struggle https://therecord.media/cybersecurity-software-liability-sta...
  • snarfy 11mo ago
    While I don't disagree, it would increase the cost of software by an extraordinary amount.
• oivey 11mo ago
  The most mind boggling number in this article to me was PeopleSoft claiming it would cost $500 million to make a payroll system for the Canadian government. That’s thousands of working years of software developers. It’s such a huge scale that it seems pretty clear the project should never start. PeopleSoft should have been dumped and the project’s scope massively reevaluated.
• Surac 11mo ago
  Please forgive me if I get something wrong. Not a native English speaker. The article boils it down to: all is a management failure. This is also my feeling after 35 years in software development. There are no such thing than a competent middle or upper management in software development. I see sometimes even devs being promoted and in an instant forget how software is made. In the other hand I see promotion of the most stupid dev. Al this leads to massive Missmanagements and hiding of problems to the upper managers. Even worse sometimes I see the best devs promoted only to watch them break because the toxin they get from there managers kills them
• Jean-Papoulos 11mo ago
  >Global IT spending has more than tripled in constant 2025 dollars since 2005, from US $1.7 trillion to $5.6 trillion, and continues to rise. Despite additional spending, software success rates have not markedly improved in the past two decades. Okay but how much more software is used ? If IT spending has tripled since 2005 but we use 10x more software I'd say the trend is good.
  • harhargange 11mo ago
    The point is not that the growth of IT spending is bad. That was just to show the scale of spending. The point of the article is that, a billion spent on software could well lead to a loss of hundred billion.
  • Yokohiii 11mo ago
    Success rates imply a ratio. Constant dollars are adjusted. Yes there is a lot more spending overall. But nothing improved quality wise, despite everyone in software somehow says they "make software better". (Which is phrased by people that don't do software, but own it.)
• bilionsnbilions 11mo ago
  > Finally, project cost-benefit justifications of software developments rarely consider the financial and emotional distress placed on end users of IT systems when something goes wrong. Most users and most management of software projects live in denial that the norm is dystopia. I can’t help think of any required and useful feature that has happened in computer usage since the early days. Easier to swallow is that the user interface of desktop operating systems hasn’t changed fundamentally in many years, yet hardware requirements continue to grow. But even the invention of a mouse requires excessive movement to move a pointer to click on something that a key combination could’ve done much more quickly. The original intention of the mouse was just as another device to use, not necessarily a primary device to direct the majority of workflow.
  • fuzzfactor 11mo ago
    From a dark storage area I may someday again get out an early Sceptre gaming monitor from the DOS days. I held on to it throughout the 1990's precisely because it was not a plug & play monitor and it was real good to install Windows with so nothing would interfere with higher resolution alternative graphics you were going to install later. Now by the 21st century it was seldom seen but these were well-made and it still worked, however the most obsolete feature that got the most interest was the sleek aftermarket plastic accessory unit attached to the side of the monitor with those sticky 3M tacky pads that are so tenacious. Yes, you've all seen it and remember it fondly, the mouse holder. Kind of like a custom cup holder that fits the standard mouse perfectly, it's obviously where you keep your mouse most of the time, except for those rare occasions when you dabble in a bit of software which actually supports a mouse. You want to keep it out of the way of your everyday desktop activities :)
• imiric 11mo ago
  The concerning aspect of all of this isn't the financial cost of these blunders, and what happened in the past. It is the increasing risk to human lives, and what will happen in the future. The Boeing case was only a sign of what's to come. Take "AI", for instance. It is being adopted left and right as if it's the solution to all of our problems, and developers, managers, and executives are increasingly relying on it. Companies and governments love it because it can cut costs and potentially make us more productive. Most people are more than happy to offload their work to it, do a cursory check of its output, if at all, and ship it or publish it and claim the work as their own. After all, it can always serve as a scapegoat if things do go wrong, and its manufacturers can brush these off as user errors. Ultimately there is no accountability. These are all components of a recipe for greater disasters. As these tools are adopted in industries where safety is paramount, in the military, etc., it's only a matter of time for more human lives to be impacted. Especially now when more egomaniacal autocrats are taking power, and surrounding themselves with yes-people. Losing face and admitting failure is not part of their playbook. We're digging ourselves into a hole we might not be able to get out of.
• a_state_full 11mo ago
  [dead]
• zelphirkalt 11mo ago
  Software development, like most other things, are part of the same make-believe market, that we run our societies in, in most countries around the world. Lets face it, most of the big money in software is believe money, not actual proven value of a thing. The word "evaluation" sort of already tells us this. It's not fact checking "How much did they sell?" or "How many users bought access or a license?", it is "How much do we believe in the future of this thing?" and risky investment "How much could we make, if this thing takes off?". For software, I am not sure this is helpful. Maybe we would develop way less trash software, if it was different. But then again we would probably still develop engagement farming software, because people would still use or buy that.
• nicholast 11mo ago
  Frederick Brooks in his essay "No Silver Bullet" (included in the collection Mythical Man Month) talked about the conventions of software development and I recall had called for taking an iterative approach to software development similar to what I had followed for the Automunge project, I went into a little more detail about that in my 2019 essay of the same name: https://medium.com/automunge/no-silver-bullet-95c77bc4bde1 https://medium.com/automunge/no-silver-bullet-95c77bc4bde1
• harhargange 11mo ago
  Why don’t i hear news of such failures from India and China?
  • t0duf0du 11mo ago
    There are great big software projects and shitty ones. IRCTC, UPI being examples of great ones. Insurance and RTO being shitty ones. I had an insurance deadline very near and the payment was not showing up in the insurance providers dashboard so had to do it twice and now it was still not showing up. Also I have faced huge problems with getting the learner's licence online. Do let me know if you've faced anything same.
    • harhargange 11mo ago
      I got my name wrong in the drivers card and never went to correct it. However most of the problems the problems there were administrative not software. I agree both irctc and upi come to mind first as the successes. Insurance, could be a particular company as i never faced such problem. Websites for Tax filing and even starting an msme has been smooth.
• pkphilip 11mo ago
  Having consulted on government projects - especially huge projects spanning dozens of government departments, what I have learnt is that the project is doomed right from the start. The specifications are written in such a way that it is impossible to get a working software which can address all of the millions (yes, literally) of specifications. For instance, I had the opportunity to review an RFP put out by a state government for software to run a single state government. The specifications stated that a SINGLE software should be used for running the full administration of all of the departments of the government - including completely disparate things such as HR, CCTV management, AI enabled monitoring of rodents and other animals near/at warehouses, all healthcare facilities, recruitment, emergency response services etc... ONE SOFTWARE for ALL of these! There isn't a single company in the world who can write software to monitor rodents, hospital appointment booking, general payroll, etc. And since the integration required was so deep, it would be impossible to use existing best-of-breed software.. and everything has to be written from scratch. How is such a software project ever going to suceeed?
  • tnolet 11mo ago
    I just fed the above into Claude Code and it one-shotted this in 5 minutes. Already doing $3B ARR after lunch.
    • fschuett 11mo ago
      Of course! This is actually very straightforward and easy, what you need is just: - One MongoDB collection (`government_stuff`) to store employees, rodents, cardiac arrest surgeries and other items as JSON - Core `/handler` API that forwards all requests to ChatGPT, figuring out if you're tracking a rodent or processing payroll - AI Vision to analyze CCTV feeds for rodents, monitors employee productivity and verify hospital equipment status - Blockchain layer for transparency (every rodent sighting minted as NFT). Estimated timeline: 2 weeks, 1 junior developer. Cost: ~$10k including token credits. Should I start implementing the main.js?
      • port11 11mo ago
        The jab at NoSQL made me snort-laugh, well done. You forgot to mention the 25 thousand npm dependencies.
• ieie3366 11mo ago
  Projects don’t fail, people do. If projects fail it means the wrong people are hired for them.
• pjmlp 11mo ago
  From consulting point of view, a common joke we use to tell, because customers demand a Ferrari, but are only willing to pay for the development costs of a Fiat.
• joduplessis 11mo ago
  I've started calling it EDD - Executive Driven Development. Senior stakeholders get into a room, decide they need xyz for the project to really succeed, push this down to managers who in turn try perform miracles with what little development resource they have. Very often they also have an offshore team who is only concerned with prolonging the contract as much as possible, rather than delivering. 2 weeks later senior stakeholders get back into the room...
  • kykat 11mo ago
    I don't know any other industry where non-technical unqualified people can just decide everything in the way that it's done with software/IT
    • Ensorceled 11mo ago
      Oh they TRY to ... it's just that the "non-technical unqualified people" get brought to heal (usually) by regulations. I've been in the room where people have tried to force a decision and a PEng, Lawyer, or CA/CPA had to say "absolutely not". It happens all the time, which is why you NEED regulations.
• bgro 11mo ago
  This is a direct result of using leetcode in interviews instead of any other, more legitimate tests like winning a tekken 1v1. Have you ever seen a good developer who’s not good at real video games? If companies had hired real developers instead of cosplayers who are stunlocked with imposter syndrome as the only candidate pool with time to memorize a rainbow table of arbitrary game trivia questions and answers, things would actually work.
• phyzix5761 11mo ago
  I wonder how much software project failure comes from lacking clear processes. Many teams, whether in companies or open source projects, never define step by step procedures for common tasks like feature development, bug triage, code review, or architectural decisions. When six developers each follow their own approach, even with consistent code style, the team can’t improve the system in a predictable and systematic way. Clear procedures don’t guarantee success, but without them teams often end up with chaos and inconsistent delivery. This lack of structured methodology seems far more common in software engineering than in other engineering disciplines.
• tonyhart7 11mo ago
  "big software project" there you are why its failling, the fact that these system is overly massive and complex that sometimes the original creator and architecture that design this system cant foresee the future needs and etc You could says its incompetence but the fact that software change so much is last 20 years make it most people cant really design a "future proof" system in a way that it cant cause trouble in the future
• ineedasername 11mo ago
  >For the foreseeable future, there are hard limits on what AI can bring to the table in controlling and managing the myriad intersections and trade-offs ?? I don’t think if thinks stopped advancing in terms of significant model improvements that actual utility would be saturated for a while. We have barely begun to consolidate potential into the tooling, use cases, knowledge sharing and depth of that knowledge throughout the workforce on how to make best use. If someone is looking at AI as a monolith in thing and thinking “oh, silver bullet to the problems of enterprise software etc” then, I really don’t know what to say except that’s on them, not on any true big claims being pushed unless you’re breaking out the long ladders to pick those cherries, or listening to people whose background and placement within things clearly makes them a bad messenger.
• gonzo41 11mo ago
  Change your idea of success to being "Propping up the consultant market", and by that definition these projects are smashing it out of the park.
• dzonga 11mo ago
  I have never seen an industry that works so hard to self-immolate. right now the industry is spending billions / trillions of $ on to train A.I on badly written open source code. in universities we teach kids - DSA - but never about to really think about scoping work nor even the unix principle of how software should compose, how to prevent errors etc. hell how many working professionals know about the 10 NASA principles & actually use them in practice ? we encourage the bright kids to go work at places which are the cathedrals of complexity - but never seeking simple solutions. & again merchants of complexity get paid more - find it easier to find jobs etc. the tragedy is the failures are documented, but also the fixes to failures as well. soon enough we're gonna raise a whole generation who doesn't know how to make reliable, robust software from scratch cz of 'vibecoding'. then ultimately civilization collapses.
  • zahlman 11mo ago
    > the 10 NASA principles Could you be more specific?
    • port11 11mo ago
      I think GP means NASA's 10 rules, but these are specifically for C development in mission-critical systems. There's been the odd software issue at NASA, but supposedly their 10 rules should make C projects much safer.
• doctorleff 11mo ago
  I taught these issues several times in the graduate Software Engineering Course. Good resources are the Standish Report: https://www.csus.edu/indiv/v/velianitis/161/chaosreport.pdf https://www.csus.edu/indiv/v/velianitis/161/chaosreport.pdf Also, anything that T. Capers Jones wrote. The most comprehensive one of these books is this: Estimating Software Costs: Bringing Realism to Estimating Hardcover ISBN-13978-0071483001 Many believe the official recognition of the crisis in developing software were the two NATO conferences in 1968 and 1969. See the Wikipedia article on the History of Software Engineering. There have been two small scale experimental comparisons of the waterfall formal model (requirements, design, code, test) and the more prototyping and agile method. They seem to have the same productivity in terms of lines per programmer-month but the formal method tends to produce larger software.
• hermitcrab 11mo ago
  >By then, the general public and the branch managers themselves finally joined Computer Weekly’s reporters (who had doggedly reported on Horizon’s problems since 2008) in the knowledge that there was something seriously wrong with Horizon’s software. Computer Weekly first broke the story, for which they deserve much credit. But I believe Private Eye did much of the long term campaigning.
• hermitcrab 11mo ago
  >not only are IT projects risky, they are the riskiest from a cost perspective. Nuclear reactor projects seem to be regularly delivered massively late and over budget.
  • OhMeadhbh 11mo ago
    Nuclear power plants usually only cost about twice as much as projected in phase II planning. IT projects are sort of open-ended. Interestingly, the simulator I was involved with (many decades ago) at a nuclear power plant came within about 10% of initial projections. The last "scan uploads for viruses" project I worked on was about 20x - 40x more expensive than projected. (Unfortunately the person who suggested we just pay a third party for this service was fired.) The bit with projecting cost and schedules for nuke plants is to ignore any initial costing and multiply the phase II planning estimate by 2 or 4.
    • hermitcrab 11mo ago
      >20x - 40x more expensive than projected That is an impressive cost overrun! I guess the issue with nuclear is that they are so expensive that even a x2 overrun is disastrous.
      • OhMeadhbh 11mo ago
        It's not as disasterous as you think. Most people paying for power plants expect large overruns. If you're the type of company that can sell bonds to get 5.5 billion dollars to build a nuclear power plant, you can sell bonds to get 11 billion dollars to build a nuclear power plant.
• trollski 11mo ago
  [dead]
• hermitcrab 11mo ago
  >Frustratingly, the IT community stubbornly fails to learn from prior failures. So far I believe that there has been too much emphasis in education on coding and algorithms (small scale/tactical stuff) and not enough emphasis on the engineering side of things like version control, QA, system design, management etc. I think the situation has changed (40 years ago most professional programmers didn't even know what version control was, let alone use vc systems) but the scope of the projects has increased faster than our skills and tools.
• hermitcrab 11mo ago
  The failed UK NHS IT project, known as the National Programme for IT (NPfIT), cost the UK government over £10 billion and produced almost nothing of value. I'm surprised that didn't get a mention. Again those bastards, Fujitsu, were involved. They even sued UK government and won £465 million settlement when their contract was cancelled. But, despite this and their complicity in covering up the failures of the Horizons posts office system, the UK government is still giving them fat contracts. If senior managers can preside over a massive failure and walk away with a huge pension, there isn't much incentive for them to do better, is there?
  • cedws 11mo ago
    The GOV.UK project is a rare success story for IT in the UK government. They need to take that team, give them big incentives to stay, and give them more projects. Why are we outsourcing to international companies that don’t give a shit when we have a wealth of talent at home? Why aren’t we investing in our own people?
    • hermitcrab 11mo ago
      The people who did the UK COVID app also did a good job, as far as I am aware. The lessons seems to be that it is better to employ a small, experienced and talented team and get out of their way, than outsource to a huge government contractor.
    • thunderfork 11mo ago
      [dead]
  • n1b0m 11mo ago
    Despite its overall failure, some parts of the infrastructure and national applications, such as the Summary Care Record and the Electronic Prescriptions Service, are considered to have survived and continue to be used
    • hermitcrab 11mo ago
      That doesn't seem much of a return on a £10 billion investment.
• hacker_yacker 11mo ago
  AI will absolutely solve these problems, by inventing nimble AI native companies that disrupt these business models into the Stone Age, worker by worker, role by role. Death by a billion cuts.
• FatherOfCurses 11mo ago
  In my very humble opinion, the impact that software has on our lives is getting to the point where software engineering should become a true profession like the other engineering branches (electrical, mechanical, etc). There should be things like professional certifications that engineers have to maintain through continuous education, a professional code of ethics, a board of review, and other functions. My reasoning is that we are at the point where a software "engineer" can make a mistake that can have the same impact as a civil engineer making a bad calculation and causing a bridge collapse. There's different levels to this, of course. An app for booking restaurant reservations wouldn't need that much oversight. But we've seen some outages having massive impacts that quite frankly did not happy twenty years ago.
• xbar 11mo ago
  Fujitsu has never been served justice for Horizon.
• marze 11mo ago
  So I haven't looked through the comments, and assume this has been discussed, but the simple solution is to limit contracts to, say, $4M, and pay only on successful completion. Then build a large project through a series of smaller steps.
• Mohsin0101 11mo ago
  [dead]
• matthewsinclair 11mo ago
  Mandatory reference to Gall’s Law [0]: > “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” [0]: https://personalmba.com/galls-law/ https://personalmba.com/galls-law/
• struc_so 11mo ago
  I feel like we lost the plot when we started optimising for 'Developer Velocity' (shipping features) over 'System Coherence' (maintainability). Most failed projects I've been around were killed by accidental complexity—too many abstraction layers wrapping simple logic.