5 ms·

Another way I've seen this put: we can already write software that's reliable the way bridges are reliable. The only problem is that people don't want to pay fo

Another way I've seen this put: we can already write software that's reliable the way bridges are reliable. The only problem is that people don't want to pay for it, wait for it, or accommodate it. Cleanroom development supposedly lead to production code with 1 bug per 10k lines back in the 80s, and NASA famously managed Shuttle code with 0 bugs in 500,000 checked lines. Other projects like Ariane 5 were not so lucky, but the general error rate is still incredibly low. To my knowledge, software in general has killed fewer spacecraft than other narrow issues like voltage inconsistencies or launch vibrations. And after the Therac-25 incident, medical device software has followed similar a development track. Even automotive software, subjected to far fewer restrictions and more varied environments, has a better critical-failure track record than well-understood components like brakes and tires. Producing software like that is orders of magnitude more expensive than producing 'normal' code. It takes far longer, in ways that more money can't neatly displace. It rarely talks to other systems, because doing so requires exhaustive testing at the interface (for every version), or testing both systems as one (with exponential testing requirements). And it's severely limited in range; freeform inputs and best-guess behavior when adjacent systems fail are out of the question. There are institutional changes that could bring us better software under existing constraints, definitely. But it's also true that the general approach to design is sort of hopeless; the sheer amount of software getting built is inconsistent with what rigorous design looks like in pretty much any field.

jrochkind1 8y ago
> Producing software like that is orders of magnitude more expensive than producing 'normal' code. Yes. Which I think points out that it's not exactly just a question of _will_ or _desire_, that we don't "want" to pay for it, wait for it, accomodate it. It would mean anyone producing software would need _an order of magnitude more money_ to do it. If we're talking about all software... it seems likely that our society literally could not afford it. I suppose we could produce an order of magnitude _less but good software_ instead. But we've built a society which is largely based on software, and been able to afford it as a society only because it's largely based on crappy software.
- Bartweiss 8y ago
  > it's not exactly just a question of _will_ or _desire_, that we don't "want" to pay for it... It would mean anyone producing software would need _an order of magnitude more money_ to do it. This is a huge point, thank you. I frequently see these discussions go down the road of personal responsibility, as in "you can't just say that it's ok for you to build things poorly because someone else would have done it if you didn't". Which is narrowly true, but strongly implies that bad code comes from individual programmers choosing to do things the easy way. The reality is that individual programmers can't choose to code this way, certainly not outside OSS. It's a structural difference that requires resources to enact. > it seems likely that our society literally could not afford it... we've built a society which is largely based on software, and been able to afford it as a society only because it's largely based on crappy software. And so there's the second major issue. We're producing software at the sort of pace associated with simple consumer goods. If you're designing a new desk organizer, you can do it without much testing, meeting basic standards like "won't fall apart quickly during normal use" and "isn't made of something poisonous to touch". An engineer might well be involved, but they're looking at issues like ease of manufacture and shipment. Someone designing a basic website or app faces the same conditions: the product will fail if it doesn't work right or is unreliable, and they might face legal trouble if it directly causes harm during normal use. I keep seeing people - including smart, influential programmers - suggest that software development lacks oversight and needs rigorous professional organizations and licenses to practice. Yonatan Zunger, as an example, draws a comparison to bridge collapses and the development of nuclear weapons, then a proposes a licensing regime like those in civil engineering, law, and medicine. (https://twitter.com/yonatanzunger/status/975545527973462016?s=21 https://twitter.com/yonatanzunger/status/975545527973462016?...) Frankly, this seems nuts to me. 99.9%+ of software development is lower-stakes than bridge design, and it has enough benefits that constraining it to the pace and price of bridge design has real downsides. I do see obvious spaces for improvement, like ensuring that there are meaningful costs for companies that lose PII while neglecting basic precautions. But if we're going to keep producing software at even a significant fraction of what we do today, there's probably no way around falling far short of 'best practices' development.
  - jrochkind1 8y ago
noir_lord 8y ago
> Producing software like that is orders of magnitude more expensive than producing 'normal' code. It takes far longer, in ways that more money can't neatly displace. Absolutely which is why the solution isn't to say "Everyone write software like NASA" it's to work on reducing the cost of writing correct software while acknowledging that the average dev is well average (and I include myself in the category of mediocrity). We need better tools and better languages that have all the safety rails possible and come with extra bubble wrap.
- Bartweiss 8y ago
  > it's to work on reducing the cost of writing correct software while acknowledging that the average dev is well average Absolutely agreed. I wrote the above because I'm frustrated by how often these conversations lead to either calls for more diligence by individual programmers (who are both average and likely working in broken systems), or draconian oversight approaches that amount to treating website developers as though they're designing bridges. The really interesting things in this domain aren't attempts to force programmers to do a better job, but to make doing a better job easier. It's stuff like Rust attempting to break new ground in safe-by-default memory management, or Let’s Encrypt lowering the bar to getting set up with TLS. Oauth has some downsides, but it's certainly helped move us away from a world where every random site makes its own sloppy attempt to manage user auth. For that matter, IDEs deserve credit for reacting to common classes of mistake; defining a local variable then operating on the input instead is the sort of thing that's easier to catch with an automated warning than with diligence or even tests. This sort of change has already taken web reliability from "sketchy at best" to "surprisingly good"; common hosts like AWS make basic load balancing and scaling straightforward, and CDNs and DDoS mitigation have become increasingly standard. We're not going to be free of bugs anytime soon, but I think there's good reason to expect actual progress.

jrochkind1 8y ago

> Producing software like that is orders of magnitude more expensive than producing 'normal' code. Yes. Which I think points out that it's not exactly just a question of _will_ or _desire_, that we don't "want" to pay for it, wait for it, accomodate it. It would mean anyone producing software would need _an order of magnitude more money_ to do it. If we're talking about all software... it seems likely that our society literally could not afford it. I suppose we could produce an order of magnitude _less but good software_ instead. But we've built a society which is largely based on software, and been able to afford it as a society only because it's largely based on crappy software.

Bartweiss 8y ago
> it's not exactly just a question of _will_ or _desire_, that we don't "want" to pay for it... It would mean anyone producing software would need _an order of magnitude more money_ to do it. This is a huge point, thank you. I frequently see these discussions go down the road of personal responsibility, as in "you can't just say that it's ok for you to build things poorly because someone else would have done it if you didn't". Which is narrowly true, but strongly implies that bad code comes from individual programmers choosing to do things the easy way. The reality is that individual programmers can't choose to code this way, certainly not outside OSS. It's a structural difference that requires resources to enact. > it seems likely that our society literally could not afford it... we've built a society which is largely based on software, and been able to afford it as a society only because it's largely based on crappy software. And so there's the second major issue. We're producing software at the sort of pace associated with simple consumer goods. If you're designing a new desk organizer, you can do it without much testing, meeting basic standards like "won't fall apart quickly during normal use" and "isn't made of something poisonous to touch". An engineer might well be involved, but they're looking at issues like ease of manufacture and shipment. Someone designing a basic website or app faces the same conditions: the product will fail if it doesn't work right or is unreliable, and they might face legal trouble if it directly causes harm during normal use. I keep seeing people - including smart, influential programmers - suggest that software development lacks oversight and needs rigorous professional organizations and licenses to practice. Yonatan Zunger, as an example, draws a comparison to bridge collapses and the development of nuclear weapons, then a proposes a licensing regime like those in civil engineering, law, and medicine. (https://twitter.com/yonatanzunger/status/975545527973462016?s=21 https://twitter.com/yonatanzunger/status/975545527973462016?...) Frankly, this seems nuts to me. 99.9%+ of software development is lower-stakes than bridge design, and it has enough benefits that constraining it to the pace and price of bridge design has real downsides. I do see obvious spaces for improvement, like ensuring that there are meaningful costs for companies that lose PII while neglecting basic precautions. But if we're going to keep producing software at even a significant fraction of what we do today, there's probably no way around falling far short of 'best practices' development.
- jrochkind1 8y ago

noir_lord 8y ago

> Producing software like that is orders of magnitude more expensive than producing 'normal' code. It takes far longer, in ways that more money can't neatly displace. Absolutely which is why the solution isn't to say "Everyone write software like NASA" it's to work on reducing the cost of writing correct software while acknowledging that the average dev is well average (and I include myself in the category of mediocrity). We need better tools and better languages that have all the safety rails possible and come with extra bubble wrap.

Bartweiss 8y ago
> it's to work on reducing the cost of writing correct software while acknowledging that the average dev is well average Absolutely agreed. I wrote the above because I'm frustrated by how often these conversations lead to either calls for more diligence by individual programmers (who are both average and likely working in broken systems), or draconian oversight approaches that amount to treating website developers as though they're designing bridges. The really interesting things in this domain aren't attempts to force programmers to do a better job, but to make doing a better job easier. It's stuff like Rust attempting to break new ground in safe-by-default memory management, or Let’s Encrypt lowering the bar to getting set up with TLS. Oauth has some downsides, but it's certainly helped move us away from a world where every random site makes its own sloppy attempt to manage user auth. For that matter, IDEs deserve credit for reacting to common classes of mistake; defining a local variable then operating on the input instead is the sort of thing that's easier to catch with an automated warning than with diligence or even tests. This sort of change has already taken web reliability from "sketchy at best" to "surprisingly good"; common hosts like AWS make basic load balancing and scaling straightforward, and CDNs and DDoS mitigation have become increasingly standard. We're not going to be free of bugs anytime soon, but I think there's good reason to expect actual progress.