13 ms·

I study and write quite a bit of tech history. IMHO from what I've learned over the last few years of this hobby, the primary issue is quite simple. While hardw

• tristor 11mo ago
  This is one part of the issue. The other major piece of this that I've seen over more than two decades in industry is that most large projects are started by and run by (but not necessarily the same person) non-technical people who are exercising political power, rather than by technical people who can achieve the desired outcomes. When you put the nexus of power into the hands of non-technical people in a technical endeavor you end up with outcomes that don't match expectations. Larger scale projects deeply suffering from "not knowing what we don't know" at the top.
  • cjbgkagh 11mo ago
    Sometimes giving people what they want can be bad for them; management wants cheap compliant workers, management gets cheap compliant workers, and then the projects fall apart in easily predictable and preventable ways. Because such failures are so common management typically isn’t punished when they do so it’s hard to keep interests inline. And because many producers are run on a cost plus basis there can be a perverse incentive to do a bad job, or at least avoid doing a good one.
  • mbesto 11mo ago
    If this were true all of the time then the fix would be simple - only have technical people in charge. My experience has shown that this (only technical people in charge) doesn't solve the problem.
    • fragmede 11mo ago
      If people didn’t work, maybe we should put an LLM in charge instead.
    • chileRick 11mo ago
      Boeing has entered the chat
    • tristor 11mo ago
      Success pretty much requires putting technical people in charge, but that doesn't mean putting technical people in charge is sufficient for success to happen. We have plenty of data over the last 40 years to prove my case. Furthermore, unfortunately, what it means to be a "technical person" is not so simple to define, unfortunately as the easy ways to codify it often exclude the very people who you want involved. Suffice to say, projects are significantly more likely to succeed when the power in the project is held by people who are competent /and/ understand the systems they are working with /and/ understand the problem domain you are developing a solution in. Whether or not they have a title like "engineer" or have a technical degree, or whatever other hallmark you might choose is largely irrelevant. What matters is competency and understanding, and then ultimately accountability. Most large projects I've been a part of or near lacked all three of these things, and thus were fundamentally doomed to failure before they ever began. The people in power lacked competency and understanding, the entire project team and the people in power lacked accountability, and competency was unevenly distributed amongst the project team. It may feel pithy, but it really is true that in many large projects the fundamental issue that leads to failure is that the decision makers don't know what they're doing and most of the implementers are incompetent. We can always root cause further to identify the incentive structures in society, and particularly in public/government projects that lead to this being true, but the fact remains at the project level this is the largest problem in my observation.
      • mbesto 11mo ago
        > Success pretty much requires putting technical people in charge, but that doesn't mean putting technical people in charge is sufficient for success to happen. This statement is terribly incongruent.
  • smokel 11mo ago
    I'm not entirely sure what you mean with "technical people" but it seems that you may not appreciate the problems that "non-technical people" try to tackle. Do your two decades of experience cover both sides?
    • tristor 11mo ago
      > Do your two decades of experience cover both sides? Yes. I appreciate both sides and have a wealth of experience in both. The challenge is that all the non-technical problems cannot be solved successfully while lacking a technical understanding. Projects generally don't fail for technical reasons, they fail because they were not set up for success, and that starts with having a clear understanding of requirements, feasibility, and a solid understanding of both the current state and the path to reach your desired outcomes, both politically/financially and technically. I was an engineer for more than a decade, I've been in Product for nearly a decade, and I'm now a senior manager in Product. I can honestly say that I have the necessary experience to hold strong opinions here and to be correct in those opinions. You need technical people who can also handle some of the non-technical aspects of a project with the reins of power if you want the project to succeed, otherwise it is doomed by the lack of understanding and competency of those in charge.
    • mring33621 11mo ago
      "you may not appreciate the problems that 'non-technical people' try to tackle." Do you mean the problem of wanting to build something without knowing how/having the skills, to build something?
• wesammikhail 11mo ago
  Agree 100%. I know a lot of people on here will disagree with me saying this but this is exactly how you get an ecosystem like javascript being as fragmented, insecure, and "trend prone" as the old school Wordpress days. It's the same problems over and over and every new "generation" of programmers has to relearn the lessons of old.
  • Salgat 11mo ago
    The difficulty lies in the fact that most software is quite cheap to generate very complex designs compared to hardware. For software designs treated similarly to hardware (such as in medical devices or at NASA), you do gain back those benefits at great expense.
• mstipetic 11mo ago
  I was so annoyed when I found out the OTP library and realized we’ve been reinventing things for 20+ years
• hackthemack 11mo ago
  I have a theory that the churn in technology is by design. If a new paradigm, new language, new framework comes out every so many years, it allows the tech sector to always want to hire new graduates for lower salaries. It gives a thin veneer of we want to always hire the person who has X when really they just do not want to hire someone with 10 years of experience in tech but who may not have picked up X yet. I do not think it is the only reason. The world is complex, but I do think it factors into why software is not treated like other engineering fields.
  • SoftTalker 11mo ago
    Constantly rewriting the same stuff in endless cycles of new frameworks and languages gives an artificial sense of productivity and justifies its own existence. If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though.
    • hackthemack 11mo ago
      I agree. But, I think the execs just say, "How can we get the most bang for our buck? If we use X, Y, Z technologies, that are the new hotness, then we will get all the new hordes of hires out there, which will make them happy, and has the added benefit of paying them less"
    • Hemospectrum 11mo ago
      > If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though. This is almost exactly what happens in some countries.
      • bdangubic 11mo ago
        which one(s)?
        • 11mo ago
• bane 11mo ago
  I've also considered a side-effect of this. Each generation of software engineers learns to operate on top of the stack of tech that came before them. This becomes their new operating floor. The generations before, when faced with a problem, would have generally achieved a solution "lower" down in the stack (or at their present baseline). But the generations today and in the future will seek to solve the problems they face on top of that base floor because they simply don't understand it. This leads to higher and higher towers of abstraction that eat up resources while providing little more functionality than if it was solved lower down. This has been further enabled by a long history of rapidly increasing compute capability and vastly increasing memory and storage sizes. Because they are only interacting with these older parts of their systems at the interface level they often don't know that problems were solved years prior, or are capable of being solved efficiently. I'm starting to see ideas that will probably form into entire pieces of software "written" on top of AI models as the new floor. Where the model basically handles all of the mainline computation, control flow, and business logic. What would have required a dozen Mhz and 4MB of RAM to run now requires TFlops and Gigabytes -- and being built from a fresh start again will fail to learn from any of the lessons learned when it was done 30 years ago and 30 layers down.
  • seeknotfind 11mo ago
    Yeah, people tend to add rather than improve. It's possible to add into lower levels without breaking things, but it's hard. Growing up as a programmer, I was taught UNUX philosophy as a golden rule, but there are sharp corners on this one: To do a new job, build afresh rather than complicate old programs by adding new "features".
    • senshan 11mo ago
      Was not it: "do one thing, do it well"?
      • bogdan 11mo ago
        There's a bit more to it https://en.wikipedia.org/wiki/Unix_philosophy https://en.wikipedia.org/wiki/Unix_philosophy
• malfist 11mo ago
  I work at $FANG, every one of our org's big projects go off the rails at the end of the project and there's always a mad rush at the end to push developers to solve all the failures of project management in their off hours before the arbitrary deadline arrives. After every single project, the org comes together to do a retrospective and ask "What can devs do differently next time to keep this from happening again". People leading the project take no action items, management doesn't hold themselves accountable at all, nor product for late changing requirements. And so, the cycle repeats next time. I led and effort one time, after a big bug made it to production after one of those crunches that painted the picture of the root cause being a huge complicated project being handed off to offshore junior devs with no supervision, and then the junior devs managing it being completely switched twice in the 8 month project with no handover, nor introspection by leadership. My manager's manager killed the document and wouldn't allow publication until I removed any action items that would constrain management. And thus, the cycle continues to repeat, balanced on the backs of developers.
  • franktankbank 11mo ago
    > wouldn't allow publication until I removed any action items that would constrain management. Thats what we call blameless culture lol
  • Sevii 11mo ago
    For how much power they have over team organization and processes, software middle management has nearly no accountability for outcomes.
    • MichaelZuo 11mo ago
      The real question is why would smart competent people continue working under management with blatant ulterior motives that negatively affect them? Why let their own credibility get dragged down for a second time, third time, fourth time, etc…? The first time is understandable but not afterwards.
      • pixelpoet 11mo ago
• pphysch 11mo ago
  There are rational explanations for this. When software fails catastrophically, people almost never die (considering how much software crashes every day). When hardware fails catastrophically, people tend to die, or lose a lot of money. There's also the complexity gap. I don't think giving someone access to the Internet Explorer codebase is necessarily going to help them build a better browser. With millions of moving parts it's impossible to tell what is essential, superfluous, high quality, low quality. Fully understanding that prior art would be a years long endeavor, with many insights no doubt, but dubious.
• mbesto 11mo ago
  I would boil this down to something else, but possibly related: project requirements are hard. That's it. > While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning. For most IT projects, software folks generally can NOT "pull apart" old systems, even if they wanted to. > Typically, software folks build new and every generation of software developers must relearn the same problems. Project management has gotten way better today than it was 20 years, so there is definitely some learnings that have been passed on.
  • rawgabbit 11mo ago
    A CIO once told me with Agile we didn’t need requirements. He thought my suggestion to document the current system before modifying was a complete waste of time. Instead he made all the developers go through a customer service workshop, how to handle and communicate with customers. Cough cough… most developers do not talk with customers. Instead where we worked developers took orders from product and project people whose titles changed every year but they operated with the mindset of a drill sergeant. My way or the highway.
    • nradov 11mo ago
      Business developers need to occasionally talk directly to customers. It's fine to filter most requirements through Product Managers / Product Owners. But developers who never communicate directly with customers and end users get disconnected from reality and end up acting based on mythology rather than ground truth.
      • rawgabbit 11mo ago
        The developers were not given the choice. They are literally mushrooms. No requirements. No interaction with customers. They took orders from product/ project people.
• alangibson 11mo ago
  "While hardware folks study and learn from the successes and failures of past hardware, software folks do not." Couldn't be further from the truth. Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult (see microservices for example). Once you've worked in both hardware and software engineering you quickly realize that they only superficially similar. Software is fundamentally philosophy, not physics. Hardware is constrained by real world limitations. Software isn't except in the most extreme cases. Result is that there is not a 'right' way to do any one thing that everyone can converge on. The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
  • Sharlin 11mo ago
    What you and the GP said are not mutually exclusive. Software engineers are quick to drink every new Kool-Aid out there, which is exactly why we’re so damned blind to history and lessons learned before.
  • jemmyw 11mo ago
    Software doesn't operate in some magical realm outside of the physical world. It very much is constrained by real world limitations. It runs on the hardware that itself is limited. I wonder if some failures are a result of thinking it doesn't have these limitations?
    • SoKamil 11mo ago
      > It very much is constrained by real world limitations. It runs on the hardware that itself is limited And yet we scale the shit out of it, shifting limitations further and further. On that scale different problems emerge and there is no single person or even single team that could comprehend this complexity in isolation. You start to encounter problems that have never been solved before.
      • jemmyw 11mo ago
• jcelerier 11mo ago
  ... are you saying that hardware projects fail less than software ones? just building a bridge is something that fails on a regular occurence all over the world. Every chip comes with list of erratas longer than my arm.
• ctkhn 11mo ago
  In my experience, a lot of the time the people who COULD be solving these issues are people who used to code or never have. The actual engineers who might do something like this aren't given authority or scope and you have MBAs or scrum masters in the way of actually solving problems.
• QuercusMax 11mo ago
  I think part of it is that reading code isn't a skill that most people are taught. When I was in grad school ages ago, my advisor told me to spend a week reading the source code of the system we were working with (TinyOS), and come back to him when I thought I understood enough to make changes and improvements. I also had a copy of the Linux Core Kernel with Commentary that I perused from time to time. Being able to dive into an unknown codebase and make sense of where the pieces are put together is a very useful skill that too many people just don't have.
  • gorbachev 11mo ago
    Being good at reading code isn't a skill that helps large software projects stay on rails. It's more about being good at juggling 1000 balls at the same time. It's 99.9% of the time a management problem, not a software problem.
    • willtemperley 11mo ago
      The successful projects I've worked on, the technical staff have been given autonomy, responsibility and full insight into the problem space. This requires managers putting a lot of trust in the engineers, but it works. Large projects I've worked on failed simply because nobody wanted the solution in the first place. In government I've seen many millions spent on projects that were either forgotten about or the politician that requested it lost office.
  • jsrcout 11mo ago
    Reading (someone else's) code is a whole lot harder than writing it. Which is unfortunate because I do an awful lot of it at work.
  • spit2wind 11mo ago
    I'm curious, what does "read code" mean to you? What does that skill look like and how is it taught?
    • onjectic 11mo ago
• RaftPeople 11mo ago
  > While hardware folks study and learn from the successes and failures of past hardware, software folks do not I've been managing, designing, building and implementing ERP type software for a long time and in my opinion the issue is typically not the software or tools. The primary issue I see is lack of qualified people managing large/complex projects because it's a rare skill. To be successful requires lots of experience and the right personality (i.e. low ego, not a person that just enjoys being in charge but rather a problem solver that is constantly seeking a better understanding). People without the proper experience won't see the landscape in front of them. They will see a nice little walking trail over some hilly terrain that extends for about a few miles. In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.
  • avemg 11mo ago
    > In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly. And boy to the people making the decisions NOT want to hear that. You'll be dismissed as a naysayer being overly conservative. If you're in a position where your words have credibility in the org, then you'll constantly be asked "what can we do to make this NOT a quest to the top of Mt Doom?" when the answer is almost always "very little".
    • Wololooo 11mo ago
      Impossible projects with impossible deadlines seems to be the norm and even when people pull them off miraculously the lesson learned is not "ok worked this time for some reason but we should not do this again", then the next people get in and go "it was done in the past why can't we do this?"
      • aakresearch 11mo ago
        Wow, sounds so familiar! I've once had to argue precisely against this very conclusion - "you saved us once in emergency, now you're bound to do it again". Wrote to my management: "It is, by all means, great when a navigator is able to take over an incapacitated pilot and make an emergency landing, thus averting the fatality. But the conclusion shouldn't be that navigators expected to perform more landings or continue to be backup pilots. Neither it should be that we completely retrain navigators as pilots and vice versa. But if navigators are assigned some extra responsibility, it should be formally acknowledged by giving them appropriate training, tools and recognition. Otherwise many written-off airplanes and hospitalized personnel would ensue." For all I know the only thing this writing might have contributed to was increased resentment by management.
• nitwit005 11mo ago
  Most of the time, there's no need to study anything. Any experienced software engineer can tell you about a project they worked on with no real requirements, management constantly changing their mind, etc.
• raincom 11mo ago
  Some consequences of NOT learning from prior successes and failures: (a) no more training for the next generation of developers/engineers (b) fighting for the best developers, and this manifests in leetcode grinding (c) decrease in cooperation among team mates, etc.
• MarcelOlsz 11mo ago
  I think this is a downstream of effect of there being no real regulation or professional designations in software which leads to every company and team being wildly different leading to no standards leaving no time for anything but crunching since there are no barriers restricting your time, so nobody spends time doing much besides shipping constantly.
• deleted 11mo ago
  [deleted]
• 01100011 11mo ago
  Software folks treat their output as if it's their baby or their art projects. Hardware folks just follow best practices and physics. They're different problem spaces though, and having done both I think HW is much simpler and easier to get right. SW is often similar if you're working on a driver or some low-level piece of code. I tried to stay in systems software throughout my career for this reason. I like doing things 'right' and don't have much need to prove to anyone how clever I am. I've met many SW folks who insist on thinking of themselves as rock stars. I don't think I've ever met a HW engineer with that attitude.
  • __mharrison__ 11mo ago
    What are the silver bullets... I mean, best practices that keep getting ignored?
  • esafak 11mo ago
    Because the software market is bigger and more competitive; hardware is mature.
• keeda 11mo ago
  I think there is a ton more nuance, but can still be explained by a simple observation, which TFA hints at: "It's the economics, stupid." Engineering is the intersection of applied sciences, economics and business. The economics aspect is almost never recognized and explains many things. Projects of other disciplines have significantly higher costs and risks, which is why they require a lot more rigor. Taking hardware as example, one bad design decision can sink the entire company. On the other hand, software has economics that span a much more diverse range than any other field. Consider: - The capital costs are extremely low. - Development can be extremely fast at the task level. - Software, once produced, can be scaled almost limitlessly for very cheap almost instantly. - The technology moves extremely fast. Most other engineering disciplines have not fundamentally changed in decades. - The technology is infinitely flexible. Software for one thing can very easily be extended for an adjacent business need. - The risks are often very low, but can be very high at the upper end. The rigor applied scales accordingly. Your LoB CRUD app going down might bother a handful of people, so who cares about tests? But your flight control software better be (and is) tested to hell and back. - Projects vary drastically in stacks, scopes and risk profiles, but the talent pool is more or less common. This makes engineering culture absolutely critical because hiring is such a crapshoot. - Extreme flexibility also masks the fact that complexity compounds very quickly. Abstractions enable elegant higher-level designs, but they mask internal details that almost always leak and introduce minor issues that cause compounding complexity. - The business rules that software automates are extremely messy to begin with (80K payroll rules!) However, the combination of a) flexibility, b) speed, and c) scalability engender a false sense of confidence. Often no attempt is made at all to simplify business requirements, which is probably where the biggest wins hide. This is also what enables requirements to shift all the time, a prime cause for failures. Worse, technical and business complexity can compound. E.g. its very easy to think "80K payroll rules linearly means O(80K) software modules" and not "wait, maybe those 80K payroll rules interact with each other, so it's probably a super-linear growth in complexity." Your architecture is then oriented towards the simplistic view, and needs hacks when business reality inevitably hits, which then start compounding complexity in the codebase. And of course, if that's a contract up for bidding, your bid is going to be unsustainably low, which will be further depressed by the competitive bidding process. If the true costs of a project -- which include human costs to the end users -- are not correctly evaluated, the design and rigor applied will be correspondingly out of whack. As such I think most failures, in addition to regular old human issues like corruption, can be attributed to an insufficient appreciation of the economics involved, driven primarily by overindexing on the powers of software without an appreciation of the pitfalls.
• smj-edison 11mo ago
  As someone who's learning programming right now, do you have any suggestions on how one would go about finding and studying these successes and failures?
  • BirAdam 11mo ago
    First, failures aren’t always obvious, and second, studying them isn’t either. This would likely need to be a formalized course. Still… If people want to know why Microsoft hated DOS and wanted to kill it with Xenix, then OS/2, then Windows, and then NT it would be vital to know that it only came about as a result of IBM wanting a 16bit source-compatible CP/M which didn’t yet exist. Then, you would likely want to read Dissecting DOS to see what limits were imposed by DOS. For other stuff, you would start backwards. Take the finished product and ask what the requirements were, then ask what the pain points are, then start digging through the source and flowcharting/mapping it. This part is a must because programs are often too difficult to really grok without some kind of map/chart. There is likely an entire discipline to be created in this…
  • wavemode 11mo ago
    The things people are talking about in this thread are less to do with the practice of programming, and more to do with the difficulties of managing (and being managed, within) an engineering organization. You'll learn all of this for yourself, on the job, just via experience.
  • t43562 11mo ago
    To be cynical, what's the point? You'll get employed and forced to be a part of them by circumstances. Your company's root priorites are probably at odds with writing good software. One Japanese company, not going to name names, kept trying to treat software as a depreciating asset. I didn't really understand well but the long and short was that fixing things that were supposed to be "done" was bad for the accounting. New things, however were good. How can you run a software company like that? But they did and got the kind of outcome you'd expect. Japan made the laws this way and gets software to match.
    • 11mo ago
• begueradj 11mo ago
  Indeed. That's why we see every now and then "new" programming paradigms which were once obsolete.
• stogot 11mo ago
  I’ve read one tech history book and I really enjoyed it. any you recommend?
  • BirAdam 11mo ago
    Hardcore Software, Fire in the Valley, Life Under the Sun… there are many.
• lifeisstillgood 11mo ago
  How do you study software history? Most of the lessons seem forever locked away behind corporate walls - any honest assessments made public will either end careers or start lawsuits
• morshu9001 11mo ago
  Yes, and it's because there aren't very many textbook ways to do software engineering, because it's evolving too fast to reach consensus.
• N_Lens 11mo ago
  Software just feels so much more ephemeral than hardware. I haven't yet met a single 'old software enthusiast' in my life, yet there are so many enthusiasts for older hardware.
  • p_v_doom 11mo ago
    I have a pet passion for an old simulation language called Dynamo. I think you will find people passionate about LISP and people that care about COBOL, and C is already multiple decades old.
  • rightbyte 11mo ago
    Doesn't retro gaming count?
  • BirAdam 11mo ago
    I am both a hardware and software enthusiast. Tons of DOS, Windows, and OS/2 software hanging around. While I don’t use them everyday, I do use them. From pre-Microsoft Visio to WordStar and MS Works for DOS, the applications are simple, powerful, and pleasing to use. While I don’t recommend anyone pull out Zenith 8bit and fire up COBOL-80 or LISP-80, they are interesting. Testing yourself in 64k is quite a challenge. The retro community is huge and varied. If it exists, someone is really into it.
• habitue 11mo ago
  This is an interesting distinction, but it ignores the reasons software engineers do that. First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc. Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.
  • BirAdam 11mo ago
    For most generations of hardware, you’re correct, but not all. For example, high-k was invented to mitigate tunneling. Sometimes, as geometries shrink, the physics involved does change.
• Yokohiii 11mo ago
  I think this is too simple. First of all, hardware people have high incentive to fully replace components and systems for many reasons. Replacement is also the only way they can fix major design mistakes. Software people constantly do fix bugs and design mistakes. There is certainly no strong culture to document or dig up former mistakes made, but it's not like they don't learn from mistakes, it's just a constant process. In contrast to hardware, there is usually no point in time to retrospect. The incentives to rejuvenate systems are low and if considered often seem expensive. Software engineers self motivation is often ill-minded, new devs feeling uncomfortable with the existing system and calling for something "modern". But if the time comes to replace the "legacy" systems, then you are right, no one looks back at the former mistakes and the devs that know them, are probably long gone. The question is whether we should ever replace an software system or focus more on gradual and active modernization. But the latter can be very hard, in hardware everything is defined, most of the time backed by standards, in software we usually don't have that, so complex interconnected systems rarely have sane upgrade paths.
• gen220 11mo ago
  IME, "Why systems fail" almost always boils down to a principal-agent problem. This is another way of expressing the Mungerism "show me the incentive, I'll show you the outcome". Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives. I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.
• aswegs8 11mo ago
  Do you mind sharing some good reads / what you wrote?