4 ms·

I don't think that lack of incentive is the primary reason that software is so unreliable. It might be part of the reason, maybe even a significant part, but no

by fhood 7y ago

I don't think that lack of incentive is the primary reason that software is so unreliable. It might be part of the reason, maybe even a significant part, but not more than that. I suspect that software is just ridiculously hard. The complexity ceiling for programming is insanely high, and there is very little tolerance for oversights. If I'm designing a drill for instance, I don't have to consider every possible use the drill will be put to. I can just overbuild the crap out of it, and cover 95% of my bases. Can you think of an analogous man made thing that is as inherently complex and fragile as a major distributed system? And importantly it never really stops. As soon as a reliable solution to a problem has been discovered we immediately use it to build a billion complex new systems on top of it. And testing is hard. And programmers rarely get to truly specialize. We spend a great deal of time learning on the go because it would be nearly impossible to truly be an expert in a field where the tools and goals change so rapidly. I'm sure I could come up with more if I kept thinking about it.

jakevoytko 7y ago
Author here! This is a good point. Earlier drafts of my post did include "the unbelievable depth and complexity of the modern technical stack" as a factor. I deferred it until a future post because I think that we accept that we don't fully understand the systems and stack because of our incentives. Imagine if the stakes were much higher: if our software was hooked up to pacemakers that stopped pumping blood when the software was down. This isn't so crazy: people who design airliners go on the test flights. We would band together as an industry to produce guidelines and recommendations around safe ways to run sites. We'd quit rather than work on a doomed project for a paycheck. Any work would require something equivalent to expert certification and design validation at every layer of the stack. We'd do everything in our power to avoid creating these emergent systems. But the situation is reversed - in many circumstances, the cost of failure couldn't be lower. We tolerate the fact that we don't understand the emergent systems that we create because it doesn't matter enough for us to care. But I don't want to make it sound like I'm dismissing you're point. You're right and it's super important. I think there's a better angle that I can use to explore this. It might be "Why do so many websites get hacked and lose our personal data?", but I feel like that doesn't fully capture it. I have to think about it. But this seems less relevant to the election case, which is what I wanted to address - that these projects fail because "launch and iterate" has become so ingrained in our process that it's difficult to recognize when it doesn't apply.
- davidw 7y ago
  I think your focus on incentives is correct. If I'm building a plane, I'm going to understand every last little piece going into it. If I'm building a web app, I'm mostly ok if I do not understand every line of Rails, and the C code in the Ruby interpreter. There are embedded systems where people have that kind of understanding. And they do fewer things than systems where people happily throw libraries in and whip out new features regularly. If I built a web app with the same kind of line by line control as that embedded system, it would get outcompeted by someone playing more 'fast and loose'. Maybe mine would have a 99.999% uptime (or maybe not, because AWS or someone might have problems too) and theirs has 99.95, but way more features, so the market will gravitate to them.
- arminiusreturns 7y ago
  This is why I have said before and will continue to say, the future of software engineering to me will largely lie in reduction of complexity, including reduction of stack complexity, reduction of LoC (a bad single metric but could be useful as part of a bigger plan) and increasing readability. I say this as an ops type who has seen stack complexity as one of the main barriers to reliability. The reasons for this being the standard are numerous and understandable though. It's essentially the main weakness of the open source ecosystems too. The many eyes theory falls apart when the LoC reach many millions. There will never be enough eyes finding bugs or poor practices. Maybe in the future machine learning could help reduce this gap.
- deleted 7y ago
  [deleted]
Wowfunhappy 7y ago
> We spend a great deal of time learning on the go because it would be nearly impossible to truly be an expert in a field where the tools and goals change so rapidly. Could we make the tools change less rapidly? I have a larger theory of the tech world right now that there's way too much change, in every corner of the industry. Computers don't have fundamentally different capabilities than they did a decade ago, so outside of security patches, why do so many software projects release major feature updates more-or-less annually? Make no mistake, most of these updates contain legitimate improvements—it's just that change itself has a high cost, for both developers and users. Most of the time, I don't think the improvements are worth the change. There are numerous cultural and business reasons for the rapid pace of change in the software world, but I don't think any of them are inherent to how software is created.
city41 7y ago
We also hand roll and rebuild things over and over again. We try to make abstractions and building blocks, but we aren't very good at it compared to other fields. It'd be like an electrician wiring up a house, and building the electrical panel from scratch every single time.
- macintux 7y ago
  Yeah, but we either need to standardize on the (metaphorical) wiring and dramatically impede progress in that area, or accept that we will indeed have to build the panel from scratch.
agumonkey 7y ago
I'm not sure even a drill has not (a lot.. ) more thoughts put into it. From mechanical to electrical engineer, to material fatigue, moulding, durability .. costs I'm no tool engineer, I'd be curious to see how it really goes at Makita or DeWalt.. how much preemptive failure mode analysis and system thinking is done even for a drill.

by fhood 7y ago

jakevoytko 7y ago

Author here! This is a good point. Earlier drafts of my post did include "the unbelievable depth and complexity of the modern technical stack" as a factor. I deferred it until a future post because I think that we accept that we don't fully understand the systems and stack because of our incentives. Imagine if the stakes were much higher: if our software was hooked up to pacemakers that stopped pumping blood when the software was down. This isn't so crazy: people who design airliners go on the test flights. We would band together as an industry to produce guidelines and recommendations around safe ways to run sites. We'd quit rather than work on a doomed project for a paycheck. Any work would require something equivalent to expert certification and design validation at every layer of the stack. We'd do everything in our power to avoid creating these emergent systems. But the situation is reversed - in many circumstances, the cost of failure couldn't be lower. We tolerate the fact that we don't understand the emergent systems that we create because it doesn't matter enough for us to care. But I don't want to make it sound like I'm dismissing you're point. You're right and it's super important. I think there's a better angle that I can use to explore this. It might be "Why do so many websites get hacked and lose our personal data?", but I feel like that doesn't fully capture it. I have to think about it. But this seems less relevant to the election case, which is what I wanted to address - that these projects fail because "launch and iterate" has become so ingrained in our process that it's difficult to recognize when it doesn't apply.

davidw 7y ago
I think your focus on incentives is correct. If I'm building a plane, I'm going to understand every last little piece going into it. If I'm building a web app, I'm mostly ok if I do not understand every line of Rails, and the C code in the Ruby interpreter. There are embedded systems where people have that kind of understanding. And they do fewer things than systems where people happily throw libraries in and whip out new features regularly. If I built a web app with the same kind of line by line control as that embedded system, it would get outcompeted by someone playing more 'fast and loose'. Maybe mine would have a 99.999% uptime (or maybe not, because AWS or someone might have problems too) and theirs has 99.95, but way more features, so the market will gravitate to them.
arminiusreturns 7y ago
This is why I have said before and will continue to say, the future of software engineering to me will largely lie in reduction of complexity, including reduction of stack complexity, reduction of LoC (a bad single metric but could be useful as part of a bigger plan) and increasing readability. I say this as an ops type who has seen stack complexity as one of the main barriers to reliability. The reasons for this being the standard are numerous and understandable though. It's essentially the main weakness of the open source ecosystems too. The many eyes theory falls apart when the LoC reach many millions. There will never be enough eyes finding bugs or poor practices. Maybe in the future machine learning could help reduce this gap.
deleted 7y ago
[deleted]

Wowfunhappy 7y ago

> We spend a great deal of time learning on the go because it would be nearly impossible to truly be an expert in a field where the tools and goals change so rapidly. Could we make the tools change less rapidly? I have a larger theory of the tech world right now that there's way too much change, in every corner of the industry. Computers don't have fundamentally different capabilities than they did a decade ago, so outside of security patches, why do so many software projects release major feature updates more-or-less annually? Make no mistake, most of these updates contain legitimate improvements—it's just that change itself has a high cost, for both developers and users. Most of the time, I don't think the improvements are worth the change. There are numerous cultural and business reasons for the rapid pace of change in the software world, but I don't think any of them are inherent to how software is created.

city41 7y ago

We also hand roll and rebuild things over and over again. We try to make abstractions and building blocks, but we aren't very good at it compared to other fields. It'd be like an electrician wiring up a house, and building the electrical panel from scratch every single time.

macintux 7y ago
Yeah, but we either need to standardize on the (metaphorical) wiring and dramatically impede progress in that area, or accept that we will indeed have to build the panel from scratch.

agumonkey 7y ago

I'm not sure even a drill has not (a lot.. ) more thoughts put into it. From mechanical to electrical engineer, to material fatigue, moulding, durability .. costs I'm no tool engineer, I'd be curious to see how it really goes at Makita or DeWalt.. how much preemptive failure mode analysis and system thinking is done even for a drill.