5 ms·

If a single simple mistake has the potential to cause devastating harm, then that is precisely the standard that should be demanded. If you do not want that, th

by Veserv 2y ago

If a single simple mistake has the potential to cause devastating harm, then that is precisely the standard that should be demanded. If you do not want that, then you can work on something where your mistakes do not cause immense harm to others or reduce the scope of the system so such harm is not possible. Imagine a mechanical engineer making industrial robots complaining about how unfair it is that they are held liable for single simple mistakes like pulping people. What injustice! If you want to make single simple mistakes, you can work on tasks where that is unlikely to cause harm like, I don't know, designing door handles? Nothing wrong with making door handles, the stakes are just different. "But I want to work on problems that can kill people (or other devastating harm), but not be responsible for killing them (or other devastating harm)" is a utterly insane position that has taken hold in software and has no place in serious, critical systems. If you can not make systems fit for the level of harm they can cause, then you have no place making such systems. That is irrespective of whether anybody in the entire world can do so; systems inadequate to minimize harm to the degree necessary to provide a positive societal cost-benefit over their lifetime are unfit for usage.

thayne 2y ago
Do you think that the mechanical engineer should be held liable if, say a criminal breaks into the factory, and sabatoges the robot to damage property or injure workers, because they didn't make the robot sufficiently resistent to sabatoge attempts?
- pjmlp 2y ago
  Moving goalposts posts, if the person was injured by something the mechanical engineer has made themselves, contrary to software industry, they are indeed liable, and can even be the target of a lawsuit. And since a mechanical engineer is a proper engineer, not someone that call themselves engineer after a six weeks bootcamp, additionally they can lose the professional title.
  - mike_hearn 2y ago
    The goalpost isn't moving. Comparisons to physical products are wrong because the question is not "are there safety critical bugs" but "can this product survive a sustained assault by highly trained adversaries", which is a standard no other product is held to.
    - pjmlp 2y ago
      Silly me thinking there is something like product recalls, food poisoning, and use cases that insurances don't cover due to high liability in hazardous goods. Anything that brings proper engineering practices into computing, and liability for malpractices, gets my vote.
      - mike_hearn 2y ago
        Software bugs almost never poison you, so that isn't applicable. Product recalls happen every day in the software industry, voluntarily. We call them software updates and the industry does them far better and more smoothly than any other industry. Use cases that aren't covered due to liability can be found in any EULA. You aren't allowed to use Windows in nuclear power stations for example.
        pjmlp 2y ago
        Actually they do, when faulty software powers something food related, anywhere on the delivery chain from the farmer to the point someone actually eats something. EULAs are meaningless in many countries outside US law, most folks are yet to bother to sue companies because they have been badly educated to accept faulty software, while they on material world, that is something that they only accept on cheap knockoffs on street bazaar, and 1 euro shops. Maybe that is the measure for software then. And even those do require a permit for selling, in many countries, or some exchange of goods between sellers and law enforcement checking permits.
    - Veserv 2y ago
      Is a non-network connected car model at risk of all cars being taken over by a economical sustained assault by highly trained adversaries in a foreign country? No. Is a network connected car vulnerable to such attacks? Yes. The product decision to make the car network connected introduced a new, previously non-existent risk. If you want to introduce that new risk, you (and anybody else who wants to introduce such risk) are responsible for the consequences. You can just not do it if that scares you. “But other people get to profit off of endangering society and I want to do it too but I do not want to be responsible for the consequences” is not a very compelling argument for letting you do that. That you need to reach higher standards for new functionality you want to add is the entire point. It is how it works in every other industry. Demanding lower standards because you want to do things that are harder and more dangerous is ass backwards.
- Veserv 2y ago
  Is that something that you expect to be attempted against a significant fraction of the products over the average expected lifetime of the product? If so, they should absolutely be required to make a product fit for its usage context that is robust against common and standard failure modes, like adversarial attacks. That such requirements are relatively uncommon in physical products due to their usage contexts being fundamentally airgapped from global adversaries is a blessing. Their requirements are lower and thus their expected standards can be lower as well. What a concept! You seem to think that standards are some kind of absolute line held in common across all product categories. Therefore it is unfair to hold some products to higher standards. That is nonsense. Expected standards are in relation to the problem and usage context and should minimize harm to the degree necessary to result in positive societal cost-benefit over their lifetime. That is how literally everything else works and principle that can be applied in general. “But my problems are different, can I ignore them and inflict devastating harm on other humans” is asinine. If you are causing net harm to society, you should not get to do it. Period.
  - thayne 2y ago
    Ok let's try another analogy. Should a carmaker be held liable if someone breaks into a car and steals something valuable (laptop, purse, etc.)? Or how about if someone drives irresponsibly (possibly under the influence) and injures or kills someone in an accident? Those are things that a significant number of cars will encounter. And yes if you don't put any effort into making your car secure or safe, there should be liability, but if you put in significant effort to security it is unreasonable to make you liable because there is more you could have done, because there is always more you could have done, and there are significant tradeoffs between security and safety and useability and cost. Maybe you could build a car that is impossible to break into, but it would be a huge pain to use, and most people wouldn't be able to afford it. > inflict devastating harm on other humans I'm not sure where you are getting this "devastating harm" from. The context of this is the large number of "urgent" security patches and vulnerability disclosures. The thing is, the vast majority of those are probably not as bad as the terminology might lead you a layperson to believe. Often in order to exploit a vulnerability in the real world, you have to chain multiple vulnerabilities together. A lot of "critical" vulnerabilities in software I need to patch, I read through the description and determine that it would be extremely difficult, or impossible to exploit the way I use the software, but the patch is still necessary because some (probably small) set of users might be using it in a way that could be exploited.
Phiwise_ 2y ago
>Imagine a mechanical engineer making industrial robots complaining about how unfair it is that they are held liable for single simple mistakes like pulping people. What injustice! If you want to make single simple mistakes, you can work on tasks where that is unlikely to cause harm like, I don't know, designing door handles? Nothing wrong with making door handles, the stakes are just different. This is a truly absurd comparison. In the first place, yes, it is in fact much easier to make physical products safer, as anyone who's ever seen a warning line or safety guard could tell you. The manufacturers of CNC mills don't accept liability by making it impossible to run a bad batch that could kill someone, they just put the whole machine in a steel box and call it a day. The software consumers want has no equivalent solution. What's worse, in the second place, these engineers aren't actually held responsible for the equivalent of most software breaches already. There pretty much is zero liability for tampering or misuse, thus the instruction booklet of 70% legal warnings that still comes with everything you buy even in this age of cutting costs. Arguing software should be held to the same standard as physical products, when software has no equivalent of safety lockouts, is just to argue it should include acceptable use sections in its terms and conditions, which is no real improvement to people's security in the face of malicious actors.

by Veserv 2y ago

thayne 2y ago

Do you think that the mechanical engineer should be held liable if, say a criminal breaks into the factory, and sabatoges the robot to damage property or injure workers, because they didn't make the robot sufficiently resistent to sabatoge attempts?

pjmlp 2y ago
Moving goalposts posts, if the person was injured by something the mechanical engineer has made themselves, contrary to software industry, they are indeed liable, and can even be the target of a lawsuit. And since a mechanical engineer is a proper engineer, not someone that call themselves engineer after a six weeks bootcamp, additionally they can lose the professional title.
- mike_hearn 2y ago
  The goalpost isn't moving. Comparisons to physical products are wrong because the question is not "are there safety critical bugs" but "can this product survive a sustained assault by highly trained adversaries", which is a standard no other product is held to.
  - pjmlp 2y ago
    Silly me thinking there is something like product recalls, food poisoning, and use cases that insurances don't cover due to high liability in hazardous goods. Anything that brings proper engineering practices into computing, and liability for malpractices, gets my vote.
    - mike_hearn 2y ago
      Software bugs almost never poison you, so that isn't applicable. Product recalls happen every day in the software industry, voluntarily. We call them software updates and the industry does them far better and more smoothly than any other industry. Use cases that aren't covered due to liability can be found in any EULA. You aren't allowed to use Windows in nuclear power stations for example.
      - pjmlp 2y ago
        Actually they do, when faulty software powers something food related, anywhere on the delivery chain from the farmer to the point someone actually eats something. EULAs are meaningless in many countries outside US law, most folks are yet to bother to sue companies because they have been badly educated to accept faulty software, while they on material world, that is something that they only accept on cheap knockoffs on street bazaar, and 1 euro shops. Maybe that is the measure for software then. And even those do require a permit for selling, in many countries, or some exchange of goods between sellers and law enforcement checking permits.
  - Veserv 2y ago
    Is a non-network connected car model at risk of all cars being taken over by a economical sustained assault by highly trained adversaries in a foreign country? No. Is a network connected car vulnerable to such attacks? Yes. The product decision to make the car network connected introduced a new, previously non-existent risk. If you want to introduce that new risk, you (and anybody else who wants to introduce such risk) are responsible for the consequences. You can just not do it if that scares you. “But other people get to profit off of endangering society and I want to do it too but I do not want to be responsible for the consequences” is not a very compelling argument for letting you do that. That you need to reach higher standards for new functionality you want to add is the entire point. It is how it works in every other industry. Demanding lower standards because you want to do things that are harder and more dangerous is ass backwards.
Veserv 2y ago
Is that something that you expect to be attempted against a significant fraction of the products over the average expected lifetime of the product? If so, they should absolutely be required to make a product fit for its usage context that is robust against common and standard failure modes, like adversarial attacks. That such requirements are relatively uncommon in physical products due to their usage contexts being fundamentally airgapped from global adversaries is a blessing. Their requirements are lower and thus their expected standards can be lower as well. What a concept! You seem to think that standards are some kind of absolute line held in common across all product categories. Therefore it is unfair to hold some products to higher standards. That is nonsense. Expected standards are in relation to the problem and usage context and should minimize harm to the degree necessary to result in positive societal cost-benefit over their lifetime. That is how literally everything else works and principle that can be applied in general. “But my problems are different, can I ignore them and inflict devastating harm on other humans” is asinine. If you are causing net harm to society, you should not get to do it. Period.
- thayne 2y ago
  Ok let's try another analogy. Should a carmaker be held liable if someone breaks into a car and steals something valuable (laptop, purse, etc.)? Or how about if someone drives irresponsibly (possibly under the influence) and injures or kills someone in an accident? Those are things that a significant number of cars will encounter. And yes if you don't put any effort into making your car secure or safe, there should be liability, but if you put in significant effort to security it is unreasonable to make you liable because there is more you could have done, because there is always more you could have done, and there are significant tradeoffs between security and safety and useability and cost. Maybe you could build a car that is impossible to break into, but it would be a huge pain to use, and most people wouldn't be able to afford it. > inflict devastating harm on other humans I'm not sure where you are getting this "devastating harm" from. The context of this is the large number of "urgent" security patches and vulnerability disclosures. The thing is, the vast majority of those are probably not as bad as the terminology might lead you a layperson to believe. Often in order to exploit a vulnerability in the real world, you have to chain multiple vulnerabilities together. A lot of "critical" vulnerabilities in software I need to patch, I read through the description and determine that it would be extremely difficult, or impossible to exploit the way I use the software, but the patch is still necessary because some (probably small) set of users might be using it in a way that could be exploited.

Phiwise_ 2y ago

>Imagine a mechanical engineer making industrial robots complaining about how unfair it is that they are held liable for single simple mistakes like pulping people. What injustice! If you want to make single simple mistakes, you can work on tasks where that is unlikely to cause harm like, I don't know, designing door handles? Nothing wrong with making door handles, the stakes are just different. This is a truly absurd comparison. In the first place, yes, it is in fact much easier to make physical products safer, as anyone who's ever seen a warning line or safety guard could tell you. The manufacturers of CNC mills don't accept liability by making it impossible to run a bad batch that could kill someone, they just put the whole machine in a steel box and call it a day. The software consumers want has no equivalent solution. What's worse, in the second place, these engineers aren't actually held responsible for the equivalent of most software breaches already. There pretty much is zero liability for tampering or misuse, thus the instruction booklet of 70% legal warnings that still comes with everything you buy even in this age of cutting costs. Arguing software should be held to the same standard as physical products, when software has no equivalent of safety lockouts, is just to argue it should include acceptable use sections in its terms and conditions, which is no real improvement to people's security in the face of malicious actors.