5 ms·

Do you think that the mechanical engineer should be held liable if, say a criminal breaks into the factory, and sabatoges the robot to damage property or injure

by thayne 2y ago

Do you think that the mechanical engineer should be held liable if, say a criminal breaks into the factory, and sabatoges the robot to damage property or injure workers, because they didn't make the robot sufficiently resistent to sabatoge attempts?

pjmlp 2y ago
Moving goalposts posts, if the person was injured by something the mechanical engineer has made themselves, contrary to software industry, they are indeed liable, and can even be the target of a lawsuit. And since a mechanical engineer is a proper engineer, not someone that call themselves engineer after a six weeks bootcamp, additionally they can lose the professional title.
- mike_hearn 2y ago
  The goalpost isn't moving. Comparisons to physical products are wrong because the question is not "are there safety critical bugs" but "can this product survive a sustained assault by highly trained adversaries", which is a standard no other product is held to.
  - pjmlp 2y ago
    Silly me thinking there is something like product recalls, food poisoning, and use cases that insurances don't cover due to high liability in hazardous goods. Anything that brings proper engineering practices into computing, and liability for malpractices, gets my vote.
    - mike_hearn 2y ago
      Software bugs almost never poison you, so that isn't applicable. Product recalls happen every day in the software industry, voluntarily. We call them software updates and the industry does them far better and more smoothly than any other industry. Use cases that aren't covered due to liability can be found in any EULA. You aren't allowed to use Windows in nuclear power stations for example.
      - pjmlp 2y ago
        Actually they do, when faulty software powers something food related, anywhere on the delivery chain from the farmer to the point someone actually eats something. EULAs are meaningless in many countries outside US law, most folks are yet to bother to sue companies because they have been badly educated to accept faulty software, while they on material world, that is something that they only accept on cheap knockoffs on street bazaar, and 1 euro shops. Maybe that is the measure for software then. And even those do require a permit for selling, in many countries, or some exchange of goods between sellers and law enforcement checking permits.
  - Veserv 2y ago
    Is a non-network connected car model at risk of all cars being taken over by a economical sustained assault by highly trained adversaries in a foreign country? No. Is a network connected car vulnerable to such attacks? Yes. The product decision to make the car network connected introduced a new, previously non-existent risk. If you want to introduce that new risk, you (and anybody else who wants to introduce such risk) are responsible for the consequences. You can just not do it if that scares you. “But other people get to profit off of endangering society and I want to do it too but I do not want to be responsible for the consequences” is not a very compelling argument for letting you do that. That you need to reach higher standards for new functionality you want to add is the entire point. It is how it works in every other industry. Demanding lower standards because you want to do things that are harder and more dangerous is ass backwards.
    - thayne 2y ago
      Ok, now consider that there is a software component, say an open source library, that has a vulnerability in it, and a car maker used that component in their network connected car, then a well funded foreign adversary exploited that vulnerability, among others, to do something nefarious. That software component wasn't specifically designed to be used in cars, but it wasn't specifically designed not to be either, it was a general purpose tool. Should the make of that component be held liable? They weren't the ones who decided to connect a car to the internet. They certainly weren't the ones who decided not to fully isolate the network-connected components from anything that could control the care. But if software doesn't have a way to push liability onto the user, then you can bet the big auto maker will sue that developer in nebraska[1] into oblivion, and the world might be worse off because of it. I don't think that you should be able to do something like sell network connected cars, without putting any effort into security. But at the same time, I think a blanket requirement to be liable for any security vulnerabilities in your software could have a lot of negative consequences. [1]: https://xkcd.com/2347/ https://xkcd.com/2347/
      - pjmlp 2y ago
        If it was an electronic component that would set the car on fire due to a short circuit, the maker of the electronic component would certainly be sued, and was the electronic component designed to be placed on car circuit boards in first place?
Veserv 2y ago
Is that something that you expect to be attempted against a significant fraction of the products over the average expected lifetime of the product? If so, they should absolutely be required to make a product fit for its usage context that is robust against common and standard failure modes, like adversarial attacks. That such requirements are relatively uncommon in physical products due to their usage contexts being fundamentally airgapped from global adversaries is a blessing. Their requirements are lower and thus their expected standards can be lower as well. What a concept! You seem to think that standards are some kind of absolute line held in common across all product categories. Therefore it is unfair to hold some products to higher standards. That is nonsense. Expected standards are in relation to the problem and usage context and should minimize harm to the degree necessary to result in positive societal cost-benefit over their lifetime. That is how literally everything else works and principle that can be applied in general. “But my problems are different, can I ignore them and inflict devastating harm on other humans” is asinine. If you are causing net harm to society, you should not get to do it. Period.
- thayne 2y ago
  Ok let's try another analogy. Should a carmaker be held liable if someone breaks into a car and steals something valuable (laptop, purse, etc.)? Or how about if someone drives irresponsibly (possibly under the influence) and injures or kills someone in an accident? Those are things that a significant number of cars will encounter. And yes if you don't put any effort into making your car secure or safe, there should be liability, but if you put in significant effort to security it is unreasonable to make you liable because there is more you could have done, because there is always more you could have done, and there are significant tradeoffs between security and safety and useability and cost. Maybe you could build a car that is impossible to break into, but it would be a huge pain to use, and most people wouldn't be able to afford it. > inflict devastating harm on other humans I'm not sure where you are getting this "devastating harm" from. The context of this is the large number of "urgent" security patches and vulnerability disclosures. The thing is, the vast majority of those are probably not as bad as the terminology might lead you a layperson to believe. Often in order to exploit a vulnerability in the real world, you have to chain multiple vulnerabilities together. A lot of "critical" vulnerabilities in software I need to patch, I read through the description and determine that it would be extremely difficult, or impossible to exploit the way I use the software, but the patch is still necessary because some (probably small) set of users might be using it in a way that could be exploited.

by thayne 2y ago

pjmlp 2y ago

Moving goalposts posts, if the person was injured by something the mechanical engineer has made themselves, contrary to software industry, they are indeed liable, and can even be the target of a lawsuit. And since a mechanical engineer is a proper engineer, not someone that call themselves engineer after a six weeks bootcamp, additionally they can lose the professional title.

mike_hearn 2y ago
The goalpost isn't moving. Comparisons to physical products are wrong because the question is not "are there safety critical bugs" but "can this product survive a sustained assault by highly trained adversaries", which is a standard no other product is held to.
- pjmlp 2y ago
  Silly me thinking there is something like product recalls, food poisoning, and use cases that insurances don't cover due to high liability in hazardous goods. Anything that brings proper engineering practices into computing, and liability for malpractices, gets my vote.
  - mike_hearn 2y ago
    Software bugs almost never poison you, so that isn't applicable. Product recalls happen every day in the software industry, voluntarily. We call them software updates and the industry does them far better and more smoothly than any other industry. Use cases that aren't covered due to liability can be found in any EULA. You aren't allowed to use Windows in nuclear power stations for example.
    - pjmlp 2y ago
      Actually they do, when faulty software powers something food related, anywhere on the delivery chain from the farmer to the point someone actually eats something. EULAs are meaningless in many countries outside US law, most folks are yet to bother to sue companies because they have been badly educated to accept faulty software, while they on material world, that is something that they only accept on cheap knockoffs on street bazaar, and 1 euro shops. Maybe that is the measure for software then. And even those do require a permit for selling, in many countries, or some exchange of goods between sellers and law enforcement checking permits.
- Veserv 2y ago
  Is a non-network connected car model at risk of all cars being taken over by a economical sustained assault by highly trained adversaries in a foreign country? No. Is a network connected car vulnerable to such attacks? Yes. The product decision to make the car network connected introduced a new, previously non-existent risk. If you want to introduce that new risk, you (and anybody else who wants to introduce such risk) are responsible for the consequences. You can just not do it if that scares you. “But other people get to profit off of endangering society and I want to do it too but I do not want to be responsible for the consequences” is not a very compelling argument for letting you do that. That you need to reach higher standards for new functionality you want to add is the entire point. It is how it works in every other industry. Demanding lower standards because you want to do things that are harder and more dangerous is ass backwards.
  - thayne 2y ago
    Ok, now consider that there is a software component, say an open source library, that has a vulnerability in it, and a car maker used that component in their network connected car, then a well funded foreign adversary exploited that vulnerability, among others, to do something nefarious. That software component wasn't specifically designed to be used in cars, but it wasn't specifically designed not to be either, it was a general purpose tool. Should the make of that component be held liable? They weren't the ones who decided to connect a car to the internet. They certainly weren't the ones who decided not to fully isolate the network-connected components from anything that could control the care. But if software doesn't have a way to push liability onto the user, then you can bet the big auto maker will sue that developer in nebraska[1] into oblivion, and the world might be worse off because of it. I don't think that you should be able to do something like sell network connected cars, without putting any effort into security. But at the same time, I think a blanket requirement to be liable for any security vulnerabilities in your software could have a lot of negative consequences. [1]: https://xkcd.com/2347/ https://xkcd.com/2347/
    - pjmlp 2y ago
      If it was an electronic component that would set the car on fire due to a short circuit, the maker of the electronic component would certainly be sued, and was the electronic component designed to be placed on car circuit boards in first place?

Veserv 2y ago

Is that something that you expect to be attempted against a significant fraction of the products over the average expected lifetime of the product? If so, they should absolutely be required to make a product fit for its usage context that is robust against common and standard failure modes, like adversarial attacks. That such requirements are relatively uncommon in physical products due to their usage contexts being fundamentally airgapped from global adversaries is a blessing. Their requirements are lower and thus their expected standards can be lower as well. What a concept! You seem to think that standards are some kind of absolute line held in common across all product categories. Therefore it is unfair to hold some products to higher standards. That is nonsense. Expected standards are in relation to the problem and usage context and should minimize harm to the degree necessary to result in positive societal cost-benefit over their lifetime. That is how literally everything else works and principle that can be applied in general. “But my problems are different, can I ignore them and inflict devastating harm on other humans” is asinine. If you are causing net harm to society, you should not get to do it. Period.

thayne 2y ago
Ok let's try another analogy. Should a carmaker be held liable if someone breaks into a car and steals something valuable (laptop, purse, etc.)? Or how about if someone drives irresponsibly (possibly under the influence) and injures or kills someone in an accident? Those are things that a significant number of cars will encounter. And yes if you don't put any effort into making your car secure or safe, there should be liability, but if you put in significant effort to security it is unreasonable to make you liable because there is more you could have done, because there is always more you could have done, and there are significant tradeoffs between security and safety and useability and cost. Maybe you could build a car that is impossible to break into, but it would be a huge pain to use, and most people wouldn't be able to afford it. > inflict devastating harm on other humans I'm not sure where you are getting this "devastating harm" from. The context of this is the large number of "urgent" security patches and vulnerability disclosures. The thing is, the vast majority of those are probably not as bad as the terminology might lead you a layperson to believe. Often in order to exploit a vulnerability in the real world, you have to chain multiple vulnerabilities together. A lot of "critical" vulnerabilities in software I need to patch, I read through the description and determine that it would be extremely difficult, or impossible to exploit the way I use the software, but the patch is still necessary because some (probably small) set of users might be using it in a way that could be exploited.