3 ms·

Author here! This is a good point. Earlier drafts of my post did include "the unbelievable depth and complexity of the modern technical stack" as a factor. I d

Author here! This is a good point. Earlier drafts of my post did include "the unbelievable depth and complexity of the modern technical stack" as a factor. I deferred it until a future post because I think that we accept that we don't fully understand the systems and stack because of our incentives. Imagine if the stakes were much higher: if our software was hooked up to pacemakers that stopped pumping blood when the software was down. This isn't so crazy: people who design airliners go on the test flights. We would band together as an industry to produce guidelines and recommendations around safe ways to run sites. We'd quit rather than work on a doomed project for a paycheck. Any work would require something equivalent to expert certification and design validation at every layer of the stack. We'd do everything in our power to avoid creating these emergent systems. But the situation is reversed - in many circumstances, the cost of failure couldn't be lower. We tolerate the fact that we don't understand the emergent systems that we create because it doesn't matter enough for us to care. But I don't want to make it sound like I'm dismissing you're point. You're right and it's super important. I think there's a better angle that I can use to explore this. It might be "Why do so many websites get hacked and lose our personal data?", but I feel like that doesn't fully capture it. I have to think about it. But this seems less relevant to the election case, which is what I wanted to address - that these projects fail because "launch and iterate" has become so ingrained in our process that it's difficult to recognize when it doesn't apply.

davidw 7y ago
I think your focus on incentives is correct. If I'm building a plane, I'm going to understand every last little piece going into it. If I'm building a web app, I'm mostly ok if I do not understand every line of Rails, and the C code in the Ruby interpreter. There are embedded systems where people have that kind of understanding. And they do fewer things than systems where people happily throw libraries in and whip out new features regularly. If I built a web app with the same kind of line by line control as that embedded system, it would get outcompeted by someone playing more 'fast and loose'. Maybe mine would have a 99.999% uptime (or maybe not, because AWS or someone might have problems too) and theirs has 99.95, but way more features, so the market will gravitate to them.
arminiusreturns 7y ago
This is why I have said before and will continue to say, the future of software engineering to me will largely lie in reduction of complexity, including reduction of stack complexity, reduction of LoC (a bad single metric but could be useful as part of a bigger plan) and increasing readability. I say this as an ops type who has seen stack complexity as one of the main barriers to reliability. The reasons for this being the standard are numerous and understandable though. It's essentially the main weakness of the open source ecosystems too. The many eyes theory falls apart when the LoC reach many millions. There will never be enough eyes finding bugs or poor practices. Maybe in the future machine learning could help reduce this gap.
deleted 7y ago
[deleted]

davidw 7y ago

I think your focus on incentives is correct. If I'm building a plane, I'm going to understand every last little piece going into it. If I'm building a web app, I'm mostly ok if I do not understand every line of Rails, and the C code in the Ruby interpreter. There are embedded systems where people have that kind of understanding. And they do fewer things than systems where people happily throw libraries in and whip out new features regularly. If I built a web app with the same kind of line by line control as that embedded system, it would get outcompeted by someone playing more 'fast and loose'. Maybe mine would have a 99.999% uptime (or maybe not, because AWS or someone might have problems too) and theirs has 99.95, but way more features, so the market will gravitate to them.

arminiusreturns 7y ago

This is why I have said before and will continue to say, the future of software engineering to me will largely lie in reduction of complexity, including reduction of stack complexity, reduction of LoC (a bad single metric but could be useful as part of a bigger plan) and increasing readability. I say this as an ops type who has seen stack complexity as one of the main barriers to reliability. The reasons for this being the standard are numerous and understandable though. It's essentially the main weakness of the open source ecosystems too. The many eyes theory falls apart when the LoC reach many millions. There will never be enough eyes finding bugs or poor practices. Maybe in the future machine learning could help reduce this gap.

deleted 7y ago

[deleted]