6 ms·

Is that something that you expect to be attempted against a significant fraction of the products over the average expected lifetime of the product? If so, they

by Veserv 2y ago

Is that something that you expect to be attempted against a significant fraction of the products over the average expected lifetime of the product? If so, they should absolutely be required to make a product fit for its usage context that is robust against common and standard failure modes, like adversarial attacks. That such requirements are relatively uncommon in physical products due to their usage contexts being fundamentally airgapped from global adversaries is a blessing. Their requirements are lower and thus their expected standards can be lower as well. What a concept! You seem to think that standards are some kind of absolute line held in common across all product categories. Therefore it is unfair to hold some products to higher standards. That is nonsense. Expected standards are in relation to the problem and usage context and should minimize harm to the degree necessary to result in positive societal cost-benefit over their lifetime. That is how literally everything else works and principle that can be applied in general. “But my problems are different, can I ignore them and inflict devastating harm on other humans” is asinine. If you are causing net harm to society, you should not get to do it. Period.

thayne 2y ago
Ok let's try another analogy. Should a carmaker be held liable if someone breaks into a car and steals something valuable (laptop, purse, etc.)? Or how about if someone drives irresponsibly (possibly under the influence) and injures or kills someone in an accident? Those are things that a significant number of cars will encounter. And yes if you don't put any effort into making your car secure or safe, there should be liability, but if you put in significant effort to security it is unreasonable to make you liable because there is more you could have done, because there is always more you could have done, and there are significant tradeoffs between security and safety and useability and cost. Maybe you could build a car that is impossible to break into, but it would be a huge pain to use, and most people wouldn't be able to afford it. > inflict devastating harm on other humans I'm not sure where you are getting this "devastating harm" from. The context of this is the large number of "urgent" security patches and vulnerability disclosures. The thing is, the vast majority of those are probably not as bad as the terminology might lead you a layperson to believe. Often in order to exploit a vulnerability in the real world, you have to chain multiple vulnerabilities together. A lot of "critical" vulnerabilities in software I need to patch, I read through the description and determine that it would be extremely difficult, or impossible to exploit the way I use the software, but the patch is still necessary because some (probably small) set of users might be using it in a way that could be exploited.
- Veserv 2y ago
  You said: “If a single simple mistake has the potential to cause devastating liability” Liability is in proportion to actual harm. A single simple mistake can only incur devastating liability if it causes devastating harm (as in harm actually occurred). If it can not and does not cause devastating harm, then you will not incur devastating liability and you are already protected. Again, you are applying some sort of bizarre uniform standard for defect remediation that ignores all nuance around intended use, intended users, usage context, harm, potential for harm, etc. That is not how anything works. Toys have different standards than planes which have different standards than bridges. The world is not unary. As for the questions of cars you posed. Those fall into slightly more nuanced principles. One of those principles is that liability is in proportion to/limited by guarantees either explicit or implicit. The limitations of the physical security of car windows are clearly communicated, obvious, and well-understood to the average consumer. A company guaranteeing that their windows are immune to being broken would be taking on the liability for their guarantee that is in excess of normal expectations. Software security and reliability is not just not clearly communicated, it is deceptively communicated. Do you believe that the average Crowdstrike customer believed that Crowdstrike could take out their business? Or were Crowdstrike’s sales and marketing continuously assuring their customers that Crowdstrike will work flawlessly despite the fine print disclaiming all liability? If the customer actually believed it could or would fuck up royally, then sure, no liability. Customer’s fault for using a product they believed to be a piece of shit. But would Crowdstrike have ever sold a single seat if their customer’s believed that? Fuck no. They made a implicit and likely explicit guarantee of operability and they are liable for it. Liability around dangerous operation is again, about clear understanding of operation guarantees. It is obvious to the most casual observer the dangers of a chainsaw and how one might be safely used. It is the job of the manufacturer to make such boundaries clear, to mitigate unknowing unsafe operation, and to mitigate unsafe operation where feasible. To go back to Crowdstrike, you can not contend that it was used “dangerously”. The defect was in failure to perform correctly, not inappropriate usage. Dangerous usage might be enabling a feature that automatically deletes any program that starts sending on the network, but you forgot that your critical server starts sending notifications to your team when under heavy load. That is likely your damages to bear. But if it was actually a feature that deletes any “anomalous program”, then that depends on the guarantees provided. Making careful tradeoffs about the distribution of damages and liability is a well-worn path. Throwing your hands up in the air because your problems are harder and more dangerous so you should be held to lower standards is absurd. And please, you make it sound like software is built to some high standard comparable to your average physical product and that it is unfair to hold it to higher standards. You and I both know software is generally held to exactly no standard. Demanding it be held to even normal standards is not some sort of unfair imposition. It should probably be held to higher standards due to the risks of correlated failures due to network connectivity as you point out, but even just getting to normal standards would be a massive and qualitative improvement.

by Veserv 2y ago

thayne 2y ago

Ok let's try another analogy. Should a carmaker be held liable if someone breaks into a car and steals something valuable (laptop, purse, etc.)? Or how about if someone drives irresponsibly (possibly under the influence) and injures or kills someone in an accident? Those are things that a significant number of cars will encounter. And yes if you don't put any effort into making your car secure or safe, there should be liability, but if you put in significant effort to security it is unreasonable to make you liable because there is more you could have done, because there is always more you could have done, and there are significant tradeoffs between security and safety and useability and cost. Maybe you could build a car that is impossible to break into, but it would be a huge pain to use, and most people wouldn't be able to afford it. > inflict devastating harm on other humans I'm not sure where you are getting this "devastating harm" from. The context of this is the large number of "urgent" security patches and vulnerability disclosures. The thing is, the vast majority of those are probably not as bad as the terminology might lead you a layperson to believe. Often in order to exploit a vulnerability in the real world, you have to chain multiple vulnerabilities together. A lot of "critical" vulnerabilities in software I need to patch, I read through the description and determine that it would be extremely difficult, or impossible to exploit the way I use the software, but the patch is still necessary because some (probably small) set of users might be using it in a way that could be exploited.

Veserv 2y ago
You said: “If a single simple mistake has the potential to cause devastating liability” Liability is in proportion to actual harm. A single simple mistake can only incur devastating liability if it causes devastating harm (as in harm actually occurred). If it can not and does not cause devastating harm, then you will not incur devastating liability and you are already protected. Again, you are applying some sort of bizarre uniform standard for defect remediation that ignores all nuance around intended use, intended users, usage context, harm, potential for harm, etc. That is not how anything works. Toys have different standards than planes which have different standards than bridges. The world is not unary. As for the questions of cars you posed. Those fall into slightly more nuanced principles. One of those principles is that liability is in proportion to/limited by guarantees either explicit or implicit. The limitations of the physical security of car windows are clearly communicated, obvious, and well-understood to the average consumer. A company guaranteeing that their windows are immune to being broken would be taking on the liability for their guarantee that is in excess of normal expectations. Software security and reliability is not just not clearly communicated, it is deceptively communicated. Do you believe that the average Crowdstrike customer believed that Crowdstrike could take out their business? Or were Crowdstrike’s sales and marketing continuously assuring their customers that Crowdstrike will work flawlessly despite the fine print disclaiming all liability? If the customer actually believed it could or would fuck up royally, then sure, no liability. Customer’s fault for using a product they believed to be a piece of shit. But would Crowdstrike have ever sold a single seat if their customer’s believed that? Fuck no. They made a implicit and likely explicit guarantee of operability and they are liable for it. Liability around dangerous operation is again, about clear understanding of operation guarantees. It is obvious to the most casual observer the dangers of a chainsaw and how one might be safely used. It is the job of the manufacturer to make such boundaries clear, to mitigate unknowing unsafe operation, and to mitigate unsafe operation where feasible. To go back to Crowdstrike, you can not contend that it was used “dangerously”. The defect was in failure to perform correctly, not inappropriate usage. Dangerous usage might be enabling a feature that automatically deletes any program that starts sending on the network, but you forgot that your critical server starts sending notifications to your team when under heavy load. That is likely your damages to bear. But if it was actually a feature that deletes any “anomalous program”, then that depends on the guarantees provided. Making careful tradeoffs about the distribution of damages and liability is a well-worn path. Throwing your hands up in the air because your problems are harder and more dangerous so you should be held to lower standards is absurd. And please, you make it sound like software is built to some high standard comparable to your average physical product and that it is unfair to hold it to higher standards. You and I both know software is generally held to exactly no standard. Demanding it be held to even normal standards is not some sort of unfair imposition. It should probably be held to higher standards due to the risks of correlated failures due to network connectivity as you point out, but even just getting to normal standards would be a massive and qualitative improvement.