5 ms·

I still think https://xkcd.com/2030/ https://xkcd.com/2030/ has to be taken seriously. You can put a whole load of verification effort into your software, which

I still think https://xkcd.com/2030/ https://xkcd.com/2030/ has to be taken seriously. You can put a whole load of verification effort into your software, which will undoubtedly make it more reliable. But you are still likely to have some kind of corner case where it breaks down. Software is complex enough for this to be universally true. The key is how we respond when the software fails. The https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 case shows an example of what not to do - when hospitals started reporting their machines giving lethal radiation doses to people, the manufacturer doubled down on the computers-are-infallible rhetoric, where they should have put every last effort into investigating. Likewise, the post office should have noticed that a rather excessive number of postmasters were apparently fiddling the books, and investigated. Instead, after it was fairly obvious that the computer was wrong, they pushed the computers-are-infallible line right through the courts, and that is what earned them the "affront to justice" judgment.

Silhouette 6y ago
The key is how we respond when the software fails. I agree, but if the first step to solving a problem is understanding that it exists then the first principle here must be to acknowledge that software systems are fallible and therefore any surprising or reasonably contested result they produce should be treated with proper caution until further information can be gathered. So many of the problems we see when modern technology goes wrong start with assuming it didn't. At that point, it's not even about how you respond to the failure, because you're denying that the failure ever happened. Big software companies with considerable lobbying power seem to be particularly good at convincing people who aren't technical experts, including most politicians, judges, juries and reporters, that this is the case. A corollary to this is that we desperately need more technological awareness among our politicians, lawyers, journalists and other relevant professions. Tech has become too big to be a minor issue you delegate to some random advisor in a basement office. It affects almost everything we do today, sometimes profoundly, and failing to understand that will inevitably lead to some horrible outcomes as we've seen all too vividly today.
Rexxar 6y ago
I agree with the sentiment but the example taken for software in this xkcd is wrong. There is a fundamental problem of trust when using software for voting systems that is not linked to the reliability of software but to the nature of voting systems and the properties we want.
- michaelt 6y ago
  Well, there are two problems and which is the fundamental one depends on your prior assumptions. Some would say it's impossible to build a secure electronic voting system, even if your supplier and their employees were completely trustworthy because between physical tampering, state-level adversaries, the state of the art in software development and the impossibility of proving a negative, such security has never been seen before. In other words, that it's an unsolvable technical problem. Others would say it's impossible to build a secure electronic voting system even if we were capable of creating flawless bug-free and tamper-proof software and hardware because the supplier will always be able to introduce undetectable bugs if they want to, and no supplier can ever be perfectly trustworthy. In other words, that it's an unsolvable social problem.
  - kosievdmerwe 6y ago
    Yeah, electronic voting is essentially like having a person in the voting booth that you have to tell your vote and trust that they will tally it correctly. [1] It doesn't matter whether voting machines are actually secure, they probably mostly are right now, but whether a layperson can have faith in the system. Paper voting is very secure if you involve people from opposing parties in the process and attacks are not very scalable. Most people can think of and understand mitigations for certain kinds of attacks. And if paper voting is too expensive for your country, you have bigger issues. [2] [1] https://www.youtube.com/watch?v=LkH2r-sNjQs https://www.youtube.com/watch?v=LkH2r-sNjQs [2] That said, I don't see how secure electronic voting can possibly be cheaper than paper voting. For voting machines to be secure, you have to manufacture them in a very audited manner, with little to no foreign sourcing of parts, you can't leave the machines unattended for long periods of time (aka, reusing them between elections is probably a no-go) and you have to build them in manner that is secure against voters tampering with them in their private booth.
zentiggr 6y ago
Something about "pride goeth before a fall" and "the one thing you can expect a manager to do is whatever shields them from liability".

Silhouette 6y ago

The key is how we respond when the software fails. I agree, but if the first step to solving a problem is understanding that it exists then the first principle here must be to acknowledge that software systems are fallible and therefore any surprising or reasonably contested result they produce should be treated with proper caution until further information can be gathered. So many of the problems we see when modern technology goes wrong start with assuming it didn't. At that point, it's not even about how you respond to the failure, because you're denying that the failure ever happened. Big software companies with considerable lobbying power seem to be particularly good at convincing people who aren't technical experts, including most politicians, judges, juries and reporters, that this is the case. A corollary to this is that we desperately need more technological awareness among our politicians, lawyers, journalists and other relevant professions. Tech has become too big to be a minor issue you delegate to some random advisor in a basement office. It affects almost everything we do today, sometimes profoundly, and failing to understand that will inevitably lead to some horrible outcomes as we've seen all too vividly today.

Rexxar 6y ago

I agree with the sentiment but the example taken for software in this xkcd is wrong. There is a fundamental problem of trust when using software for voting systems that is not linked to the reliability of software but to the nature of voting systems and the properties we want.

michaelt 6y ago
Well, there are two problems and which is the fundamental one depends on your prior assumptions. Some would say it's impossible to build a secure electronic voting system, even if your supplier and their employees were completely trustworthy because between physical tampering, state-level adversaries, the state of the art in software development and the impossibility of proving a negative, such security has never been seen before. In other words, that it's an unsolvable technical problem. Others would say it's impossible to build a secure electronic voting system even if we were capable of creating flawless bug-free and tamper-proof software and hardware because the supplier will always be able to introduce undetectable bugs if they want to, and no supplier can ever be perfectly trustworthy. In other words, that it's an unsolvable social problem.
- kosievdmerwe 6y ago
  Yeah, electronic voting is essentially like having a person in the voting booth that you have to tell your vote and trust that they will tally it correctly. [1] It doesn't matter whether voting machines are actually secure, they probably mostly are right now, but whether a layperson can have faith in the system. Paper voting is very secure if you involve people from opposing parties in the process and attacks are not very scalable. Most people can think of and understand mitigations for certain kinds of attacks. And if paper voting is too expensive for your country, you have bigger issues. [2] [1] https://www.youtube.com/watch?v=LkH2r-sNjQs https://www.youtube.com/watch?v=LkH2r-sNjQs [2] That said, I don't see how secure electronic voting can possibly be cheaper than paper voting. For voting machines to be secure, you have to manufacture them in a very audited manner, with little to no foreign sourcing of parts, you can't leave the machines unattended for long periods of time (aka, reusing them between elections is probably a no-go) and you have to build them in manner that is secure against voters tampering with them in their private booth.

zentiggr 6y ago

Something about "pride goeth before a fall" and "the one thing you can expect a manager to do is whatever shields them from liability".